Skip to content

Lau 7 structured founder review generation - #302

Open
ctrl-peace wants to merge 34 commits into
mainfrom
lau-7-structured-founder-review-generation
Open

Lau 7 structured founder review generation#302
ctrl-peace wants to merge 34 commits into
mainfrom
lau-7-structured-founder-review-generation

Conversation

@ctrl-peace

@ctrl-peace ctrl-peace commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

Added a deterministic benchmark and evaluation framework for Founder Weekly Review generation quality.

Includes:

  • Evaluator logic for validating generated reviews against evidence snapshots.
  • Benchmark cases covering citations, source types, unsupported claims, contradictory evidence, malformed payloads, and empty evidence.
  • Benchmark runner and baseline report generation.

Testing

Ran:

pnpm eval:founder-weekly-review

The benchmark currently runs against fixture-generated payloads and does not yet run through the current Founder Weekly Review generation implementation.

Baseline:

  • 19 benchmark cases
  • 19 passing cases
  • Overall score: 0.482

Current Gaps

The baseline identified:

  • Invalid or missing citations in generated claims.
  • Incorrect source type usage (ex: founder context treated as customer feedback).
  • Evidence coverage and semantic matching improvements needed.
  • Unsupported generalizations and shipped claims needing stronger validation.

Future work should integrate the benchmark with the actual Founder Weekly Review generation flow to measure real generation performance.

EricLiu2795 and others added 17 commits July 19, 2026 10:57
Remove the legacy plaintext embedding credential columns from the
company Drizzle schema and drop the runtime fallback reads that still
queried them. Migration 0011 already drops those columns, so clean
migration-replayed databases failed on full company selects/returning.

Move the backfill script under apps/web and switch it to raw SQL so it
can still migrate older db:push-shaped databases that retain the
columns.

Co-authored-by: Kien Le <kien-ship-it@users.noreply.github.com>
Restore the missing Changesets config and record a patch release note
for the @launchstack/core schema/runtime alignment with migration 0011.

Co-authored-by: Kien Le <kien-ship-it@users.noreply.github.com>
Assert the company Drizzle schema and credential read path stay aligned
with migration 0011, and add an optional live DB check that select/insert
succeed after the plaintext embedding columns are dropped.

Co-authored-by: Kien Le <kien-ship-it@users.noreply.github.com>
@vercel

vercel Bot commented Aug 1, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
launch-stack Error Error Aug 8, 2026 11:58pm
pdr-ai-v2 Error Error Aug 8, 2026 11:58pm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants