Add ET vision kernels for SmolVLM2#10
Closed
AFOliveira wants to merge 127 commits into
Closed
Conversation
This commit adds a stub implementation of the ET backend. Doesn't do anything beside creating the infrastructure that can be later expanded to effectively implement a backend. It builds, registers the backend and finds no devices at initialisation time.
Device with offboard memory is considered GPU. ACCEL type is for accelerators like Blas.
This change initializes the runtime, and in the process fixes the initialization of the backend in general. Currently only supported operation is listing of devices.
GUID is a required property and is checked very early during initialization.
Buffer usage is decided during model loading, before any interrogation for ops support. Even if ops are not supported it should be possible to allocate/set_tensor/get_tensor on device buffers. Until we implement device buffer management we need to declare host buffer support, so that fallback to CPU backend works correctly.
The integer device used by ggml for device id is now called devidx. This will help separate it more clearly from the _runtime_ id.
For now, we can't get the amount of free memory. This will be fixed in the future through modifications to runtime. We use free == total, _for_ _now_.
This will help us, in the future, to create a better description thanks to the runtime device properties.
This will be very useful in future calls to the runtime, to identify our device.
Allocate reg before initialising devices so that the 'reg' pointer is valid.
- Allocate/free memory on device - Implement host-to-device and device-to-host memory transfers
ggml checks existence of these functions and calls then instead of synchronous ones thus braking inference (specifically it needs to transfer logits from device to host for sampling).
Enhance ET backend and sync upstream llama.cpp changes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add ET backend coverage for the three operations required to keep the
SmolVLM2/SigLIP vision graph entirely on ET-SoC1:
IM2COLfor patch embeddingsNORM)UNARY)The backend advertises only the supported type, layout, alignment, and unary
variant combinations. Each operation is implemented as an embedded ET device
kernel and dispatched by the existing graph executor.
Validation
test-backend-ops test -b ET -o NORM,UNARY,IM2COL: 54/54 supported LayerNorm/IM2COL cases pass.test-backend-ops test -b ET -o GELU: supported F32 GELU case passes.The end-to-end six-frame benchmark improved from a 4.7837 to 4.6711 real-time
factor while removing all vision CPU fallbacks.