-
Notifications
You must be signed in to change notification settings - Fork 0
feat(core): support AICORE_SERVICE_KEY env var for credentials #58
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -9,7 +9,7 @@ | |||||||||||
|
|
||||||||||||
| from ai_core_sdk.helpers import get_home | ||||||||||||
| from ai_core_sdk.helpers.constants import (AI_CORE_PREFIX, AUTH_ENDPOINT_SUFFIX, CONFIG_FILE_ENV_VAR, PROFILE_ENV_VAR, | ||||||||||||
| VCAP_AICORE_SERVICE_NAME, VCAP_SERVICES_ENV_VAR) | ||||||||||||
| SERVICE_KEY_ENV_VAR, VCAP_AICORE_SERVICE_NAME, VCAP_SERVICES_ENV_VAR) | ||||||||||||
| from ai_core_sdk.helpers.logging import get_logger | ||||||||||||
|
|
||||||||||||
| logger = get_logger() | ||||||||||||
|
|
@@ -241,6 +241,37 @@ def _str_or_none(value) -> Optional[str]: | |||||||||||
| return str(value) if value else None | ||||||||||||
|
|
||||||||||||
|
|
||||||||||||
| def _parse_service_key(credential_values: List[CredentialsValue]) -> Optional[Callable[[CredentialsValue], Optional[str]]]: | ||||||||||||
| """Return a source getter for AICORE_SERVICE_KEY if the env var is set and valid JSON, else None. | ||||||||||||
|
|
||||||||||||
| AICORE_SERVICE_KEY is expected to be the raw JSON string of a BTP service key, i.e. the | ||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [pp] Not entirely. Service key and the |
||||||||||||
| ``credentials`` object from a VCAP_SERVICES binding without the outer envelope. Credential | ||||||||||||
| fields are extracted using the ``vcap_key`` paths already defined on each ``CredentialsValue``, | ||||||||||||
| but with the leading ``'credentials'`` segment stripped (same as the CLI's load_service_key). | ||||||||||||
| """ | ||||||||||||
| raw = os.environ.get(SERVICE_KEY_ENV_VAR) | ||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [req] |
||||||||||||
| if not raw: | ||||||||||||
| return None | ||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It could be an option to not return here but write an empty dict into service_key and then return a Callable instead of Optional[Callable]. Could make the code below more consistent. |
||||||||||||
| try: | ||||||||||||
| service_key = json.loads(raw) | ||||||||||||
| except json.JSONDecodeError as exc: | ||||||||||||
| raise ValueError( | ||||||||||||
| f"{SERVICE_KEY_ENV_VAR} is set but contains invalid JSON: {exc}" | ||||||||||||
| ) from exc | ||||||||||||
|
|
||||||||||||
| def _get(cv: CredentialsValue) -> Optional[str]: | ||||||||||||
| if not cv.vcap_key: | ||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [req] (maybe for the future) After struggling and looking into the codebase, I finally understand what is a I know it is not super relevant to this PR. But refactoring needed... And allow me to just put my thoughts here as a reference. I added this comment to the refactoring BLI. I think we should really consider renaming few stuff.
Also it is pretty hard to understand what And why do we define object schema in such a complicated way. Can we not just define the data class of the @mwien to give you an overview of how {
"VCAP_SERVICES": {
"aicore": [
{
"label": "aicore",
"provider": null,
"plan": "<plan>",
"name": "default_aicore",
"tags": [],
"instance_guid": "<some uuid>",
"instance_name": "default_aicore",
"binding_guid": "<some_other_uuid>",
"binding_name": null,
"credentials": {
"serviceurls": {
"AI_API_URL": "https://api.ai.......ml.hana.ondemand.com"
},
"appname": "<appname>",
"clientid": "<clientid>",
"clientsecret": "<clientsecret>",
"identityzone": "<subdomain of the subaccount>",
"identityzoneid": "<tenant id / subaccount id>",
"url": "<auth_url>"
},
"syslog_drain_url": null,
"volume_mounts": []
}
],
// ...
}
} |
||||||||||||
| return None | ||||||||||||
| # vcap_key is e.g. ('credentials', 'clientid') — drop the 'credentials' prefix | ||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [pp] I had a hard time understanding what this comment was trying to tell me. Information that helped me understand:
This might be partially me as a python noob, but consider sharpening the wording a bit (at least something like that: "prefix" => leading "credentials" element). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
What about this. |
||||||||||||
| key_path = cv.vcap_key[1:] | ||||||||||||
| try: | ||||||||||||
| return _str_or_none(get_nested_value(service_key, key_path)) | ||||||||||||
| except KeyError: | ||||||||||||
| return None | ||||||||||||
|
|
||||||||||||
| return _get | ||||||||||||
|
|
||||||||||||
|
|
||||||||||||
| def fetch_credentials(profile: str = None, credential_values: List[CredentialsValue] = CORE_CREDENTIAL_VALUES, | ||||||||||||
| validate: bool = True, **kwargs) -> Dict[str, str]: | ||||||||||||
| """ | ||||||||||||
|
|
@@ -261,11 +292,17 @@ def fetch_credentials(profile: str = None, credential_values: List[CredentialsVa | |||||||||||
| except KeyError: | ||||||||||||
| vcap_service = None | ||||||||||||
|
|
||||||||||||
| service_key_getter = _parse_service_key(credential_values) | ||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If the change suggested in line 304 is done, this could be moved to an inline call to _parse_service_key there |
||||||||||||
|
|
||||||||||||
| sources = [ | ||||||||||||
| Source("kwargs", | ||||||||||||
| lambda cv: _str_or_none(kwargs.get(cv.name))), | ||||||||||||
| Source("environment variables", | ||||||||||||
| lambda cv: _str_or_none(os.environ.get(f'{AI_CORE_PREFIX}_{cv.name.upper()}'))), | ||||||||||||
| *( | ||||||||||||
| [Source(SERVICE_KEY_ENV_VAR, service_key_getter)] | ||||||||||||
| if service_key_getter is not None else [] | ||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe an option to keep this more consistent would be to have service_key_getter be a Callable instead of Optional[Callable] and have it always return None in case the env var doesn't exist. See comment above. |
||||||||||||
| ), | ||||||||||||
| Source("config file", | ||||||||||||
| lambda cv: _str_or_none(config.get(f'{AI_CORE_PREFIX}_{cv.name.upper()}'))), | ||||||||||||
| Source("VCAP service", | ||||||||||||
|
|
||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,6 +8,7 @@ | |
| DEFAULT_HOME_PATH = os.path.join(os.path.expanduser('~'), '.aicore') | ||
| HOME_PATH_ENV_VAR = f'{AI_CORE_PREFIX}_HOME' | ||
| PROFILE_ENV_VAR = f'{AI_CORE_PREFIX}_PROFILE' | ||
| SERVICE_KEY_ENV_VAR = f'{AI_CORE_PREFIX}_SERVICE_KEY' | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [pp] I think There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [q/pp] I see that Maybe a follow-up: Overall I find the current constant list a bit too complex. I understand the idea behind "having constant strings defined as variable to avoid typo" methodology here. Perhaps follow what React does: Similar to using
It improves the auto completion experience. Also I would prefer inline It doesn't have to be part of this PR though. But since we might have a lot of things needing refactoring, I will create a refactoring BLI and collect some ideas incl. this one. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Btw, some old tradition for PR review: q: Question (Something I am not sure about) Some of us this kind of form to indicate the type of review comments for better communication. |
||
| VCAP_AICORE_SERVICE_NAME = 'aicore' | ||
| VCAP_SERVICES_ENV_VAR = 'VCAP_SERVICES' | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -96,6 +96,7 @@ | |
| <strong>DEFAULT_HOME_PATH</strong> = '/home/runner/.aicore'<br> | ||
| <strong>HOME_PATH_ENV_VAR</strong> = 'AICORE_HOME'<br> | ||
| <strong>PROFILE_ENV_VAR</strong> = 'AICORE_PROFILE'<br> | ||
| <strong>SERVICE_KEY_ENV_VAR</strong> = 'AICORE_SERVICE_KEY'<br> | ||
| <strong>VCAP_AICORE_SERVICE_NAME</strong> = 'aicore'<br> | ||
| <strong>VCAP_SERVICES_ENV_VAR</strong> = 'VCAP_SERVICES'</td></tr></table> | ||
| </body></html> | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [pp] Consider not making unnecessary changes. Also it is a good practice to always leave an empty line at the end of a file (coming from a unix world). |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -37,10 +37,17 @@ dev = [ | |
| "pyhamcrest==2.1.0", | ||
| "pytest-dotenv>=0.5.2", | ||
| ] | ||
| docs = [ | ||
| "sphinx<9.0.0", | ||
| "sphinxawesome-theme", | ||
| ] | ||
|
Comment on lines
+40
to
+43
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [req] this is unrelated to this change, right? |
||
|
|
||
| [tool.pytest.ini_options] | ||
| testpaths = ["tests"] | ||
| norecursedirs = ["integration_tests"] | ||
| # Prevent pytest-dotenv from loading the repo-root .env (which contains real credentials) | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [q/req] Is there anywhere documented that we should store Later we will have a sample server folder / package, which will be used for trying out purposes and we can put |
||
| # into unit tests. Integration tests load it explicitly via conftest. | ||
| env_files = [] | ||
|
|
||
| [project.scripts] | ||
| aicore = "ai_core_sdk.cli:cli" | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -15,7 +15,7 @@ | |||||
| init_conf, CORE_CREDENTIAL_VALUES, | ||||||
| ) | ||||||
| from ai_core_sdk.helpers.constants import (AI_CORE_PREFIX, HOME_PATH_ENV_VAR, PROFILE_ENV_VAR, VCAP_SERVICES_ENV_VAR, | ||||||
| VCAP_AICORE_SERVICE_NAME, CONFIG_FILE_ENV_VAR) | ||||||
| VCAP_AICORE_SERVICE_NAME, CONFIG_FILE_ENV_VAR, SERVICE_KEY_ENV_VAR) | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [pp] Not related to this PR, but possibly a nice new backlog item: I noticed that the variables are imported in a different order here. So this does not seem to be covered by linting. Possibly worth improving, if you think so too. |
||||||
|
|
||||||
| VCAP_SERVICE_DICT = { | ||||||
| VCAP_AICORE_SERVICE_NAME: [{ | ||||||
|
|
@@ -290,6 +290,77 @@ def test_init_conf_permission_error(self, mock_logger): | |||||
| # Restore permissions for cleanup in teardown | ||||||
| config_file.chmod(0o644) | ||||||
|
|
||||||
| @patch('ai_core_sdk.credentials.logger') | ||||||
| def test_fetch_credentials_from_service_key(self, mock_logger): | ||||||
| mock_logger.debug = MagicMock() | ||||||
|
|
||||||
| service_key = { | ||||||
| 'clientid': 'sk-client-id', | ||||||
| 'clientsecret': 'sk-client-secret', | ||||||
| 'url': 'https://sk-auth-url', | ||||||
| 'serviceurls': {'AI_API_URL': 'https://sk-api-url'}, | ||||||
| } | ||||||
| with patch.dict(os.environ, {SERVICE_KEY_ENV_VAR: json.dumps(service_key)}): | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [req] I believe this should be
Suggested change
Also, if I am not mistaken, why do the tests pass? 🧐 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @marikaner It was also quite confusing to me.. If I understand it right, I suggested above to change those constant names in the future. At least to something like |
||||||
| credentials = fetch_credentials() | ||||||
|
|
||||||
| self.assertEqual(credentials['client_id'], 'sk-client-id') | ||||||
| self.assertEqual(credentials['client_secret'], 'sk-client-secret') | ||||||
| self.assertEqual(credentials['auth_url'], 'https://sk-auth-url/oauth/token') | ||||||
| self.assertEqual(credentials['base_url'], 'https://sk-api-url/v2') | ||||||
| mock_logger.debug.assert_any_call(f"Using credentials from: {SERVICE_KEY_ENV_VAR}") | ||||||
|
|
||||||
| @patch('ai_core_sdk.credentials.logger') | ||||||
| def test_fetch_credentials_from_service_key_x509(self, mock_logger): | ||||||
| mock_logger.debug = MagicMock() | ||||||
|
|
||||||
| service_key = { | ||||||
| 'clientid': 'sk-client-id', | ||||||
| 'certurl': 'https://sk-cert-url', | ||||||
| 'certificate': 'sk-cert-content', | ||||||
| 'key': 'sk-key-content', | ||||||
| 'serviceurls': {'AI_API_URL': 'https://sk-api-url'}, | ||||||
| } | ||||||
| with patch.dict(os.environ, {SERVICE_KEY_ENV_VAR: json.dumps(service_key)}): | ||||||
| credentials = fetch_credentials() | ||||||
|
|
||||||
| self.assertEqual(credentials['client_id'], 'sk-client-id') | ||||||
| self.assertEqual(credentials['cert_str'], 'sk-cert-content') | ||||||
| self.assertEqual(credentials['key_str'], 'sk-key-content') | ||||||
| self.assertEqual(credentials['auth_url'], 'https://sk-cert-url/oauth/token') | ||||||
| self.assertEqual(credentials['base_url'], 'https://sk-api-url/v2') | ||||||
| mock_logger.debug.assert_any_call(f"Using credentials from: {SERVICE_KEY_ENV_VAR}") | ||||||
|
|
||||||
| @patch('ai_core_sdk.credentials.logger') | ||||||
| def test_service_key_lower_precedence_than_env_vars(self, mock_logger): | ||||||
| mock_logger.debug = MagicMock() | ||||||
|
|
||||||
| service_key = { | ||||||
| 'clientid': 'sk-client-id', | ||||||
| 'clientsecret': 'sk-client-secret', | ||||||
| 'url': 'https://sk-auth-url', | ||||||
| 'serviceurls': {'AI_API_URL': 'https://sk-api-url'}, | ||||||
| } | ||||||
| with patch.dict(os.environ, { | ||||||
| SERVICE_KEY_ENV_VAR: json.dumps(service_key), | ||||||
| f'{AI_CORE_PREFIX}_CLIENT_ID': 'env-client-id', | ||||||
| f'{AI_CORE_PREFIX}_CLIENT_SECRET': 'env-client-secret', | ||||||
| f'{AI_CORE_PREFIX}_AUTH_URL': 'https://env-auth-url', | ||||||
| f'{AI_CORE_PREFIX}_BASE_URL': 'https://env-base-url', | ||||||
|
Comment on lines
+347
to
+348
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [pp] If you're not checking those, maybe it is not needed to add them here? |
||||||
| }): | ||||||
| credentials = fetch_credentials() | ||||||
|
|
||||||
| # env vars win | ||||||
| self.assertEqual(credentials['client_id'], 'env-client-id') | ||||||
| self.assertEqual(credentials['client_secret'], 'env-client-secret') | ||||||
| mock_logger.debug.assert_any_call("Using credentials from: environment variables") | ||||||
|
|
||||||
| def test_service_key_invalid_json_raises(self): | ||||||
| with patch.dict(os.environ, {SERVICE_KEY_ENV_VAR: 'not-valid-json'}): | ||||||
| with self.assertRaises(ValueError) as ctx: | ||||||
| fetch_credentials() | ||||||
| self.assertIn(SERVICE_KEY_ENV_VAR, str(ctx.exception)) | ||||||
| self.assertIn('invalid JSON', str(ctx.exception)) | ||||||
|
|
||||||
| @patch('ai_core_sdk.credentials.logger') | ||||||
| def test_injecting_credential_values(self, mock_logger): | ||||||
| mock_logger.debug = MagicMock() | ||||||
|
|
||||||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[req] The function is called
_parse_service_key. Based on its name, nobody would expect it to return a callable. Function name should really be speaking. The actual json parsing part is like a precondition to the function.Also
credential_valuesis never used. This function is readingSERVICE_KEY_ENV_VARfrom env.This function is more like
create_service_key_getteror something.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[req] Consider separating the JSON parsing / validation part out into a different function.