Skip to content

Sandbox Support - #118

Merged
mmedici-rf merged 68 commits into
mainfrom
malware
Aug 11, 2026
Merged

Sandbox Support#118
mmedici-rf merged 68 commits into
mainfrom
malware

Conversation

@ebartosevic

Copy link
Copy Markdown
Contributor

No description provided.

@hudson-woomer
hudson-woomer self-requested a review July 20, 2026 00:29
Comment thread psengine/sandbox/client.py
Co-authored-by: hudson-woomer <hudson.woomer@recordedfuture.com>

@hudson-woomer hudson-woomer left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PSF-1194 — Malware Config - add token in the config attributes

One issue to fix:

sandbox_token=None → opaque AttributeError (client.py:80). config.sandbox_token.get_secret_value() has no None guard, so Config.init(sandbox_token=None), a config.json with "sandbox_token": null, or Config.init(sandbox_token=os.environ.get('X')) when X is unset all raise AttributeError before the intended ValueError('Missing … Sandbox API token.'). Unlike rf_token (which has a mode='before' validator coercing None → ''), the new field skipped that normalization. Fix: add the same mode='before' validator for sandbox_token.

Minor/optional: client.py:36 uses re.match for the token format check, so a trailing newline slips past it; re.fullmatch (like asi/client.py) would reject it.

Everything else looks good — the token is wired through correctly (reaches the Authorization: Bearer header, validated against the 40-hex shape, excluded from save_config, masked in logs), consistent with asi_token.

Comment thread psengine/sandbox/client.py
Comment thread psengine/sandbox/client.py Outdated

@hudson-woomer hudson-woomer left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PSF-1312 — profile endpoints (GET/POST /profiles, GET/PUT/DELETE /profiles/{id})

Everything looks good — a few recommended changes:

  • fetch_profiles doesn't paginate and crashes on an empty tenant (sandbox_mgr.py:1025-1027). It makes a single request and reads only response.json()['data']: (1) it ignores the next cursor, so a company with more than one page of profiles silently gets only page 1; (2) when a company has no profiles the API returns data: null (not []), so for e in None raises an unwrapped TypeError. Fixed both by routing it through request_paged (like fetch_samples/search_samples) and null-guarding request_paged's first page (page = json_response['data'] or []).
  • Profile id/name isn't URL-encoded(fetch_profile/update_profile/delete_profile), and all three accept a name — a name with #///? misroutes the request, so delete_profile('prod#legacy') hits /profiles/prod and deletes the wrong profile. Encoded it with quote(profile_id, safe='.') (as fusion/entity_match/enrich already do).
  • Closed network/browser Literals on the response model → one enum value the API adds later (it omits safari, for instance) makes fetch_profiles raise and return zero profiles. Widened the response types to NetworkMode | str / Browser | str.
  • create_profile/update_profile don't validate geolocation-requires-vpn locally the way submit_sample does, so the bad combo only fails as a round-trip 400 instead of a clear up-front error. Added the same model_validator.
  • example_6.py pins a fixed profile name → if a prior live-doc-CI run dies before its cleanup delete, the leftover makes the next run's create 409 and CI stays red. Pre-deleted the name first so it delete tolerates 404.
  • "all fields except id must be submitted" contradicts the next sentence (omitted optional fields are cleared) and the optional geolocation/browser.

Comment thread psengine/sandbox/sandbox_mgr.py
mocker.patch.object(sandbox_mgr.sb_client, 'request', return_value=mock)

assert sandbox_mgr.fetch_profiles() == []

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def test_fetch_profiles_null_data(self, sandbox_mgr: SandboxMgr, mocker, make_response):
# GET /profiles returns data: null (not []) when the company has no profiles.
mock = make_response({'data': None, 'next': None})
mocker.patch.object(sandbox_mgr.sb_client, 'request', return_value=mock)
assert sandbox_mgr.fetch_profiles() == []

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test for GET /profiles where a company has no profiles.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Closed network/browser Literals on the response model → one enum value the API adds later (it omits safari, for instance) makes fetch_profiles raise and return zero profiles. Widened the response types to NetworkMode | str / Browser | str.

This has been rolled back. The point of the model Ernest made is to validate only those specific browsers/network modes. Adding | str invalidate the point of having a literal Browser. We can consider to only validate str | None rather than Browser | None, but having Browser | str | None is just pointless

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the test_fetch_profiles_null_data actually a real test? meaning, have you seen the API returning null on data?

Comment thread psengine/sandbox/sandbox_mgr.py
Comment thread psengine/sandbox/sandbox_mgr.py Outdated
Comment thread psengine/sandbox/sandbox_mgr.py Outdated
Comment thread psengine/sandbox/sandbox.py Outdated
Comment thread psengine/sandbox/client.py Outdated
Comment thread psengine/sandbox/sandbox.py
Comment thread docs/modules/sandbox.md Outdated
Comment thread docs/examples/sandbox/example_6.py Outdated
hudson-woomer

This comment was marked as off-topic.

Changes requested within scope of PSF-1190

Co-authored-by: hudson-woomer <hudson.woomer@recordedfuture.com>

@hudson-woomer hudson-woomer left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PSF-1190 review

@hudson-woomer hudson-woomer left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PSF-1194
Reviewed sandbox module additions. Found 5 issues: two fixed (BehavioralTask fields made optional to handle in-progress API responses, repeated-offset debug log restored in request_paged), three outstanding (results_per_page: int | None crash, fetch_profiles null crash + missing pagination, re.match vs re.fullmatch inconsistency in token validation).

Comment thread psengine/sandbox/client.py
kind: Literal['behavioral']

tags: list[str] = []
score: int

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
score: int
score: int | None = None

PSF-1194
For the following few changes on below lines match every other task model in the file and OverviewTask


tags: list[str] = []
score: int
target: str

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
target: str
target: str | None = None

tags: list[str] = []
score: int
target: str
backend: str

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
backend: str
backend: str | None = None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you find any samples that are failing for this? or any docs that specified they can be null?

score: int
target: str
backend: str
resource: str

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
resource: str
resource: str | None = None

Comment thread psengine/config/config.py Outdated
offset = json_response.get('next')
prev_offset = None

while offset and offset != prev_offset and len(all_results) < max_results:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
while offset and offset != prev_offset and len(all_results) < max_results:
while offset and len(all_results) < max_results:
if offset == prev_offset:
self.log.debug(f'Paged request returned a repeated offset {offset!r}; stopping.')
break

offset != prev_offset prevents infinite loop but diagnostic log was dropped during the tidy-up.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hudson-woomer I am confused, probably due to the very bad Github UI 😞

This is what i see in that part of the code:

        while offset and offset != prev_offset and len(all_results) < max_results:
            params['offset'] = offset
            params['limit'] = min(results_per_page, max_results - len(all_results))
            response = self.request(
                method=method,
                url=url,
                headers=headers,
                data=data,
                params=params,
                **kwargs,
            )
            json_response = response.json()
            page = json_response.get('data', [])

            if not page:
                # The last page is always empty
                self.log.debug('Paged request returned an empty `data` page; stopping.')
                break

            all_results.extend(page)
            prev_offset = offset
            offset = json_response.get('next')

        return all_results[:max_results]

what is exactly your suggested change?

@mmedici-rf

Copy link
Copy Markdown
Collaborator

@hudson-woomer sorry for the amount of tags, I think the branch is more or less ok to be merged. There are a few comments left on your changes when you have time. Thanks!

@mmedici-rf

Copy link
Copy Markdown
Collaborator

@hudson-woomer i am going to dismiss the changes requested since the PR has been waiting for answers for 2 weeks. Feel free to raise a new PR, when this is merged

@mmedici-rf
mmedici-rf dismissed hudson-woomer’s stale review August 11, 2026 12:35

i am going to dismiss the changes requested since the PR has been waiting for answers for 2 weeks. Feel free to raise a new PR, when this is merged

@mmedici-rf
mmedici-rf merged commit 41fbf83 into main Aug 11, 2026
9 checks passed
@mmedici-rf
mmedici-rf deleted the malware branch August 11, 2026 12:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants