Sandbox Support - #118
Conversation
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: hudson-woomer <hudson.woomer@recordedfuture.com>
There was a problem hiding this comment.
PSF-1194 — Malware Config - add token in the config attributes
One issue to fix:
sandbox_token=None → opaque AttributeError (client.py:80). config.sandbox_token.get_secret_value() has no None guard, so Config.init(sandbox_token=None), a config.json with "sandbox_token": null, or Config.init(sandbox_token=os.environ.get('X')) when X is unset all raise AttributeError before the intended ValueError('Missing … Sandbox API token.'). Unlike rf_token (which has a mode='before' validator coercing None → ''), the new field skipped that normalization. Fix: add the same mode='before' validator for sandbox_token.
Minor/optional: client.py:36 uses re.match for the token format check, so a trailing newline slips past it; re.fullmatch (like asi/client.py) would reject it.
Everything else looks good — the token is wired through correctly (reaches the Authorization: Bearer header, validated against the 40-hex shape, excluded from save_config, masked in logs), consistent with asi_token.
hudson-woomer
left a comment
There was a problem hiding this comment.
PSF-1312 — profile endpoints (GET/POST /profiles, GET/PUT/DELETE /profiles/{id})
Everything looks good — a few recommended changes:
fetch_profilesdoesn't paginate and crashes on an empty tenant (sandbox_mgr.py:1025-1027). It makes a single request and reads onlyresponse.json()['data']: (1) it ignores thenextcursor, so a company with more than one page of profiles silently gets only page 1; (2) when a company has no profiles the API returnsdata: null(not[]), sofor e in Noneraises an unwrappedTypeError. Fixed both by routing it throughrequest_paged(likefetch_samples/search_samples) and null-guardingrequest_paged's first page (page = json_response['data'] or []).- Profile id/name isn't URL-encoded(
fetch_profile/update_profile/delete_profile), and all three accept a name — a name with#///?misroutes the request, sodelete_profile('prod#legacy')hits/profiles/prodand deletes the wrong profile. Encoded it withquote(profile_id, safe='.')(as fusion/entity_match/enrich already do). - Closed
network/browserLiterals on the response model → one enum value the API adds later (it omitssafari, for instance) makesfetch_profilesraise and return zero profiles. Widened the response types toNetworkMode | str/Browser | str. create_profile/update_profiledon't validate geolocation-requires-vpnlocally the waysubmit_sampledoes, so the bad combo only fails as a round-trip 400 instead of a clear up-front error. Added the samemodel_validator.example_6.pypins a fixed profile name → if a prior live-doc-CI run dies before its cleanup delete, the leftover makes the next run's create 409 and CI stays red. Pre-deleted the name first so it delete tolerates 404.- "all fields except
idmust be submitted" contradicts the next sentence (omitted optional fields are cleared) and the optionalgeolocation/browser.
| mocker.patch.object(sandbox_mgr.sb_client, 'request', return_value=mock) | ||
|
|
||
| assert sandbox_mgr.fetch_profiles() == [] | ||
|
|
There was a problem hiding this comment.
| def test_fetch_profiles_null_data(self, sandbox_mgr: SandboxMgr, mocker, make_response): | |
| # GET /profiles returns data: null (not []) when the company has no profiles. | |
| mock = make_response({'data': None, 'next': None}) | |
| mocker.patch.object(sandbox_mgr.sb_client, 'request', return_value=mock) | |
| assert sandbox_mgr.fetch_profiles() == [] |
There was a problem hiding this comment.
Test for GET /profiles where a company has no profiles.
There was a problem hiding this comment.
Closed network/browser Literals on the response model → one enum value the API adds later (it omits safari, for instance) makes fetch_profiles raise and return zero profiles. Widened the response types to NetworkMode | str / Browser | str.
This has been rolled back. The point of the model Ernest made is to validate only those specific browsers/network modes. Adding | str invalidate the point of having a literal Browser. We can consider to only validate str | None rather than Browser | None, but having Browser | str | None is just pointless
There was a problem hiding this comment.
Is the test_fetch_profiles_null_data actually a real test? meaning, have you seen the API returning null on data?
Changes requested within scope of PSF-1190 Co-authored-by: hudson-woomer <hudson.woomer@recordedfuture.com>
hudson-woomer
left a comment
There was a problem hiding this comment.
PSF-1194
Reviewed sandbox module additions. Found 5 issues: two fixed (BehavioralTask fields made optional to handle in-progress API responses, repeated-offset debug log restored in request_paged), three outstanding (results_per_page: int | None crash, fetch_profiles null crash + missing pagination, re.match vs re.fullmatch inconsistency in token validation).
| kind: Literal['behavioral'] | ||
|
|
||
| tags: list[str] = [] | ||
| score: int |
There was a problem hiding this comment.
| score: int | |
| score: int | None = None |
PSF-1194
For the following few changes on below lines match every other task model in the file and OverviewTask
|
|
||
| tags: list[str] = [] | ||
| score: int | ||
| target: str |
There was a problem hiding this comment.
| target: str | |
| target: str | None = None |
| tags: list[str] = [] | ||
| score: int | ||
| target: str | ||
| backend: str |
There was a problem hiding this comment.
| backend: str | |
| backend: str | None = None |
There was a problem hiding this comment.
did you find any samples that are failing for this? or any docs that specified they can be null?
| score: int | ||
| target: str | ||
| backend: str | ||
| resource: str |
There was a problem hiding this comment.
| resource: str | |
| resource: str | None = None |
| offset = json_response.get('next') | ||
| prev_offset = None | ||
|
|
||
| while offset and offset != prev_offset and len(all_results) < max_results: |
There was a problem hiding this comment.
| while offset and offset != prev_offset and len(all_results) < max_results: | |
| while offset and len(all_results) < max_results: | |
| if offset == prev_offset: | |
| self.log.debug(f'Paged request returned a repeated offset {offset!r}; stopping.') | |
| break |
offset != prev_offset prevents infinite loop but diagnostic log was dropped during the tidy-up.
There was a problem hiding this comment.
@hudson-woomer I am confused, probably due to the very bad Github UI 😞
This is what i see in that part of the code:
while offset and offset != prev_offset and len(all_results) < max_results:
params['offset'] = offset
params['limit'] = min(results_per_page, max_results - len(all_results))
response = self.request(
method=method,
url=url,
headers=headers,
data=data,
params=params,
**kwargs,
)
json_response = response.json()
page = json_response.get('data', [])
if not page:
# The last page is always empty
self.log.debug('Paged request returned an empty `data` page; stopping.')
break
all_results.extend(page)
prev_offset = offset
offset = json_response.get('next')
return all_results[:max_results]what is exactly your suggested change?
Co-authored-by: hudson-woomer <hudson.woomer@recordedfuture.com>
|
@hudson-woomer sorry for the amount of tags, I think the branch is more or less ok to be merged. There are a few comments left on your changes when you have time. Thanks! |
|
@hudson-woomer i am going to dismiss the changes requested since the PR has been waiting for answers for 2 weeks. Feel free to raise a new PR, when this is merged |
i am going to dismiss the changes requested since the PR has been waiting for answers for 2 weeks. Feel free to raise a new PR, when this is merged
No description provided.