-
Notifications
You must be signed in to change notification settings - Fork 0
LLM chat module #116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
LLM chat module #116
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
Repository: edge-toolkit/core
Length of output: 197
🏁 Script executed:
Repository: edge-toolkit/core
Length of output: 21718
🏁 Script executed:
Repository: edge-toolkit/core
Length of output: 50373
🏁 Script executed:
Repository: edge-toolkit/core
Length of output: 41452
🌐 Web query:
Transformers.js 4.2.0 onnxruntime-web 1.24.3 or 1.27.0 wasmPaths ort-wasm-simd-threaded.jsep WebGPU💡 Result:
Transformers.js v4.2.0 is a significant release that introduced a new WebGPU runtime rewritten in C++ and deeper integration with ONNX Runtime [1][2]. Compatibility between
@huggingface/transformersand onnxruntime-web is critical, as version mismatches often lead to initialization errors such as "no available backend found" [3]. Key Technical Details: 1. Version Compatibility: It is recommended to use the exact version of onnxruntime-web that is specified in the package.json file of the@huggingface/transformersrelease you are using [3]. While Transformers.js 4.2.0 dependencies shift over time, it is designed to work with the specific ONNX Runtime builds it was tested against [1][2]. 2. WebGPU and JSEP: The WebGPU Execution Provider (EP) in ONNX Runtime Web utilizes the JavaScript Execution Provider (JSEP) architecture [4]. When using WebGPU, ONNX Runtime typically requires the jsep-enabled WASM binaries, specifically ort-wasm-simd-threaded.jsep.wasm [4][5]. 3. Configuring wasmPaths: If you are hosting the WASM files yourself (rather than using the defaults provided by the library), you must correctly configure ort.env.wasm.wasmPaths [4]. When using WebGPU, ensure your path configuration points to the directory containing the jsep-enabled binaries [4][5]: ort.env.wasm.wasmPaths = 'https://your-cdn.com/path/to/wasm/files/'; Note that modern ONNX Runtime versions may require specific filenames (e.g., ort-wasm-simd-threaded.jsep.wasm) to be available at the path provided [5]. If using a CDN like JSDelivr or UNPKG, you can point to the version-specific directory of onnxruntime-web [4]. 4. WebGPU Initialization: To leverage WebGPU, ensure your environment supports it (e.g., Chrome/Edge 113+ on supported hardware) [6]. Within your application, you may need to explicitly import the WebGPU-enabled entry point or specify 'webgpu' in your execution provider settings [7][8]: import * as ort from 'onnxruntime-web/webgpu'; const session = await ort.InferenceSession.create(modelPath, { executionProviders: ['webgpu'] }); If you encounter issues, verify that your build tool (like Webpack or Vite) is not accidentally aliasing or bundling conflicting versions of onnxruntime-web [3]. [9][10][1][3][4][5][7][6][8][2]Citations:
🏁 Script executed:
Repository: edge-toolkit/core
Length of output: 147
🌐 Web query:
site:github.com/huggingface/transformers.js "ort-wasm-simd-threaded.jsep" "wasmPaths"💡 Result:
The term ort-wasm-simd-threaded.jsep refers to a specific variant of the ONNX Runtime Web (ORT) WebAssembly binary, typically used to enable SIMD (Single Instruction, Multiple Data) and JSEP (JavaScript Execution Provider) support for improved performance [1][2]. When developing with ONNX Runtime Web—particularly through libraries like
@huggingface/transformers—you may encounter issues loading these binary files if your environment requires a specific location or if you are working in a restricted context (e.g., local development, offline apps, or environments with strict CORS/CSP policies) [3][4][5]. The wasmPaths configuration is the mechanism used to tell the runtime where to locate these.wasm and.mjs files [6][4]. Key Technical Points: - Configuration: You can override the default CDN-based loading (which often points to jsDelivr) by setting the wasmPaths property in the ONNX Runtime environment configuration [6][5]. For example: env.backends.onnx.wasm.wasmPaths = '/path/to/your/wasm/files/'; [4] - Local Hosting: To serve these files locally, copy the necessary.wasm and.mjs files from your node_modules/onnxruntime-web/dist/ folder into your project's public or assets directory [4][7]. - CORS and COEP: Loading these files from a different origin or via blob URLs can trigger CORS or COEP (Cross-Origin-Embedder-Policy) errors [3][8]. Ensure your server is configured with proper headers (e.g., Cross-Origin-Opener-Policy: same-origin and Cross-Origin-Embedder-Policy: require-corp) if you are using multi-threading (SharedArrayBuffer) [3][1]. - Caching and Performance: For production or offline applications, consider pre-fetching the binary and assigning it to env.backends.onnx.wasm.wasmBinary [6][8]. Using a pre-set wasmBinary can also bypass certain dynamic loading issues related to blob URLs and import.meta.url [8][9]. If you are experiencing errors such as "Failed to fetch dynamically imported module," it usually indicates that the application is trying to load the file from an incorrect path or that the browser's security policy is blocking the request [4][10]. Always ensure the version of the.wasm file matches the version of onnxruntime-web installed in your project [10][5].Citations:
loadWasmFactoryhuggingface/transformers.js#1532🌐 Web query:
onnxruntime-web 1.27.0 env.ts wasmPaths jsep webgpu ort-wasm-simd-threaded.jsep.mjs💡 Result:
In ONNX Runtime Web 1.27.0, the
ort-wasm-simd-threaded.jsep.mjsfile and its corresponding.wasmbinary are required when using the JavaScript Execution Provider (JSEP), which powers the WebGPU and WebNN execution providers [1][2]. The library attempts to locate these files automatically based on the location of your JavaScript bundle [1][2]. If your WebAssembly files are located in a different directory or hosted on a CDN, you must use theort.env.wasm.wasmPathsconfiguration to override the default paths [3][1][2]. Configuration Options You can setort.env.wasm.wasmPathsin two ways: 1. As a string (path prefix): If you provide a string, the library appends the default filenames (e.g.,ort-wasm-simd-threaded.jsep.wasm) to that prefix [3][1]. ort.env.wasm.wasmPaths = 'https://cdn.jsdelivr.net/npm/onnxruntime-web@1.27.0/dist/'; 2. As an object (explicit overrides): If you need to specify individual files, you can provide an object mapping the expected filenames to their absolute URLs [3]. ort.env.wasm.wasmPaths = { 'ort-wasm-simd-threaded.jsep.wasm': 'https://example.com/path/to/ort-wasm-simd-threaded.jsep.wasm', 'ort-wasm-simd-threaded.jsep.mjs': 'https://example.com/path/to/ort-wasm-simd-threaded.jsep.mjs' }; Important Notes - Absolute Paths: When usingwasmPaths, the provided URLs must be absolute [4][5]. - Automatic Switching: Note that settingwasmPathsdoes not automatically cause ONNX Runtime to switch to the JSEP-enabled runtime [5]. You must ensure the environment is configured to use the appropriate execution provider (e.g., WebGPU) when creating your inference session [3]. - File Consistency: Theort-wasm-simd-threaded.jsep.mjsfile is the JavaScript glue code required to load the JSEP-enabled WebAssembly binary [6]. Always ensure the.mjsand.wasmfiles from the same build version are used together [7][8].Citations:
Stage the JSEP ORT assets for the WebGPU path.
The
webgpuexecution provider uses the JSEP runtime. Copyort-wasm-simd-threaded.jsep.{mjs,wasm}topkg/ort/; otherwise pipeline initialization fails when ORT requests these files.🤖 Prompt for AI Agents