Skip to content

USER_DATA_STORAGE.md

Latest commit

 

History

History
213 lines (175 loc) · 5.21 KB

USER_DATA_STORAGE.md

File metadata and controls

213 lines (175 loc) · 5.21 KB
Raw

User Data Storage System

Overview

The bot now uses Telegram's context storage system instead of a local SQLite database for user-specific data. This ensures:

✅ Per-user isolation - Each user's data is completely separate ✅ Easy data management - Users can clear their data with /cleardata ✅ No data mixing - Multiple users' data never overlaps ✅ Persistent storage - Data is preserved between bot restarts

How It Works

Storage Mechanism

  • Uses context.user_data dictionary for each user
  • Automatically persists to telegram_user_data/bot_persistence.pkl
  • Each user's data is indexed by their Telegram user ID
  • Bot-wide stats (total users) stored in context.bot_data

What's Stored Per User

context.user_data = {
    'initialized': True,
    'bookmarks': [
        {
            'url': 'https://example.com',
            'title': 'Example Site',
            'created_at': '2026-01-24T...',
            'last_scan': '2026-01-24T...',
            'last_status': 'AVAILABLE'
        }
    ],
    'scan_history': [
        {
            'url': 'https://example.com',
            'results': {...},
            'timestamp': '2026-01-24T...'
        }
    ],
    'preferences': {
        'show_detailed': True,
        'auto_bookmark': False,
        'notification_enabled': True
    },
    'first_seen': '2026-01-24T...',
    'last_seen': '2026-01-24T...'
}

Commands

/start

  • Initializes user data structure if first time
  • Increments total user count
  • Shows real-time status with:
    • Bot uptime
    • Total registered users
    • Scanner count

/cleardata

  • Shows confirmation dialog
  • Permanently deletes all user data:
    • Bookmarks
    • Scan history
    • Preferences
  • Reinitializes fresh data structure

/bookmarks

  • Lists user's bookmarks
  • Shows status indicators (🟢/🔴)
  • Data retrieved from context.user_data['bookmarks']

/stats

  • Shows user statistics
  • Calculates from context.user_data
  • No database queries needed

/settings

  • Manages user preferences
  • Stored in context.user_data['preferences']

Data Privacy

Isolation

  • Each user's data is completely separate
  • No cross-user data access possible
  • User IDs are the only identifier

Clearing Data

When user runs /cleardata:

  1. All their data is deleted from memory
  2. Fresh data structure is created
  3. No trace of previous data remains
  4. Persistence file is updated automatically

Storage Location

  • File: telegram_user_data/bot_persistence.pkl
  • Excluded from git via .gitignore
  • Only accessible by bot process
  • Not shared between bot instances

Technical Details

Persistence System

persistence = PicklePersistence(
    filepath='telegram_user_data/bot_persistence.pkl',
    store_data={
        'user_data': True,   # Per-user data
        'bot_data': True,    # Bot-wide stats
        'chat_data': False,  # Not used
        'callback_data': False
    }
)

Automatic Saving

  • Data is saved automatically by python-telegram-bot
  • No manual save operations needed
  • Updates persist immediately

Data Limits

  • Scan history limited to last 100 scans per user
  • Prevents memory bloat
  • Automatically trims oldest entries

Migration from Database

Removed

  • ❌ Local SQLite database for user data
  • ❌ users table
  • ❌ bookmarks table
  • ❌ scan_history table
  • ❌ user_preferences table

Kept

  • ✅ Database class (for future bot-wide stats if needed)
  • ✅ Bot initialization and shutdown

Benefits

  1. Simpler architecture - No database management
  2. Better isolation - Built-in per-user separation
  3. User control - Easy data deletion
  4. Portable - Single pickle file
  5. No SQL - Pure Python data structures

User Experience

What Users See

  • Real-time user count in /start
  • Personal data accessible only to them
  • Clear data control with /cleardata
  • Fast operations (no database queries)

What Users Control

  • Their bookmarks
  • Their scan history
  • Their preferences
  • Complete data deletion

Security

Access Control

  • Only bot has access to persistence file
  • User data indexed by Telegram user ID
  • No external database access needed
  • No SQL injection risks

Data Retention

  • Data persists until user clears it
  • No automatic deletion
  • User controls their data lifecycle
  • Bot restart preserves all data

Backup

To backup user data:

# Copy persistence file
cp telegram_user_data/bot_persistence.pkl telegram_user_data/backup_$(date +%Y%m%d).pkl

To restore:

# Restore from backup
cp telegram_user_data/backup_20260124.pkl telegram_user_data/bot_persistence.pkl

Troubleshooting

Data not persisting

  • Check telegram_user_data/ directory exists
  • Verify write permissions
  • Check bot shutdown gracefully

User lost data

  • Verify persistence file exists
  • Check file not corrupted
  • User may have run /cleardata

Memory issues

  • Each user limited to 100 scans
  • Old scans automatically trimmed
  • Consider periodic cleanup task

Summary

The new storage system provides:

  • ✅ Complete user data isolation
  • ✅ Easy data management for users
  • ✅ No data mixing between users
  • ✅ Simple architecture
  • ✅ Better privacy control
  • ✅ Persistent storage via Telegram