Skip to content

Change the Canarytokens fetch method to retrieve tokens in batches #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
thinkst-tom wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Change the Canarytokens fetch method to retrieve tokens in batches #27

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2f8d931
Change the canarytokens fetch method so that it hits the paginate end…
thinkst-tom Jun 22, 2026
0771cc2
Simplify code
thinkst-tom Jul 3, 2026
7cf1016
PR comments
thinkst-tom Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions canarytools/models/canarytokens.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
import logging
logger = logging.getLogger('canarytools')

CANARYTOKENS_PAGE_SIZE_LIMIT = 2500

@thinkst-taariq thinkst-taariq Jul 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be an optional parameter of CanaryTokens.all?

I'm not saying it should, I'm just wondering what the pros and cons are.

@thinkst-tom thinkst-tom Jul 6, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to enforce a standard limit, with testing based on what our default instance size can handle. So we wouldn't want to allow a page size bigger than this value.

2500 is also big enough to minimise the amount of calls necessary for a Console with >>>2500 tokens.

I also don't think a caller of the CanaryTokens.all method needs to know that the paginate endpoint is being called.

@thinkst-taariq thinkst-taariq Jul 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Thanks for the reasoning. That helps clear some stuff up for me.


class CanaryTokens(object):
def __init__(self, console):
"""Initialize CanaryToken
Expand Down Expand Up @@ -137,8 +139,30 @@ def all(self, include_endpoints=True):
>>> import canarytools
>>> tokens = console.tokens.all()
"""
params = {'include_endpoints': str(include_endpoints)}
return self.console.get('canarytokens/fetch', params, self.parse)
cursor = None
all_tokens = []

while True:
params = {
'limit': CANARYTOKENS_PAGE_SIZE_LIMIT,
'render_transients': str(include_endpoints),
}

if cursor:
params['cursor'] = cursor

response = self.console.get('canarytokens/paginate', params, parser=dict)

all_tokens.extend(
CanaryToken.parse(self.console, token)
for token in response.get('canarytokens', [])
)

cursor = response.get('cursor', {})
if not cursor:
break
Comment thread
thinkst-taariq marked this conversation as resolved.

return all_tokens

def parse(self, data):
"""Parse JSON data
Expand Down