TeiaCareInferenceClient

Welcome to TeiaCareInferenceClient!

TeiaCareInferenceClient is a C++ inference client library that implements KServe protocol.

Getting Started

This project uses git submodules so it is required to clone it using the --recursive flag in order to retrive the required submodules.

git clone https://github.com/TeiaCare/TeiaCareInferenceClient.git --recursive

This project relies on venvpp2 (included as the scripts submodule) to manage the development environment based on Conan v2 and CMake v4 with CMakePresets as first class citizens.

Create Development Environment

Run the venvpp2 setup script from the project root. It creates a .venv at the project root, pins CONAN_HOME to ./.conan2 and installs the Python requirements (Conan, CMake, Ninja, pre-commit, etc.).

# Linux/MacOS
scripts/scripts/env/setup.sh

# Windows
scripts\scripts\env\setup.bat

Start Development Environment

On subsequent sessions just re-activate the virtual environment created above.

# Linux/MacOS
source .venv/bin/activate

# Windows
.venv\Scripts\activate.bat

Setup Build Environment (Windows Only)

When building from command line on Windows it is necessary to activate the Visual Studio Developer Command Prompt.

Depending on the version of Visual Studio compiler and on its install location it is required to run vcvars64.bat script the set the development environment properly.

Note: using Visual Studio IDE or the CMake extension for VSCode this step is already managed in the background, so no action is required.

Examples:

# Visual Studio 2022 - Build Tools
"C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvars64.bat"

# Visual Studio 2019 - Enterprise
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\Build\vcvars64.bat"

Configure Conan Remote

Configure the TeiaCare Artifactory remote to fetch 3rd party dependencies (only required for unit tests, examples and benchmarks).

conan remote add teiacare https://artifactory.app.teiacare.com/artifactory/api/conan/teiacare --index 0 --force
conan remote login teiacare <ARTIFACTORY_USERNAME> -p <ARTIFACTORY_PASSWORD>

Install Dependencies

Install Conan packages and generate CMakePresets.json together with the CMake toolchain file. Pick the profile that matches your platform (available profiles are under .ci/profiles/, the same ones used by CI): linux-clang-15, linux-gcc-12, macos-arm64-clang15, windows-msvc-2022.

# Linux (clang) - Debug
conan install inference_client -b=missing -pr:a=.ci/profiles/linux-clang-15 -s build_type=Debug

# Linux (gcc) - Release
conan install inference_client -b=missing -pr:a=.ci/profiles/linux-gcc-12 -s build_type=Release

# Windows (msvc) - Release
conan install inference_client -b=missing -pr:a=.ci/profiles/windows-msvc-2022 -s build_type=Release

Configure, Build and Install

Use the CMake presets generated by Conan to configure, build and install the library.

# CMake configure
cmake --preset conan-debug

# CMake build
cmake --build build/Debug

# CMake install
cmake --install build/Debug

Replace conan-debug / build/Debug with conan-release / build/Release when building the Release configuration.

Examples

# Build all the examples
python scripts/cmake.py <Debug|Release|RelWithDebInfo> <COMPILER_NAME> <COMPILER_VERSION> --examples --warnings

# Run all the examples
python scripts/tools/run_examples.py install/examples

Examples are installed in $PWD/install/examples.

Unit Tests and Code Coverage

# Build Unit Tests with Code Coverage enabled (if supported)
python scripts/cmake.py <Debug|Release|RelWithDebInfo> <COMPILER_NAME> <COMPILER_VERSION> --coverage --warnings

# Run Unit Tests
python scripts/tools/run_unit_tests.py <Debug|Release|RelWithDebInfo>

# Run Code Covergae
python scripts/tools/run_coverage.py <COMPILER_NAME> <COMPILER_VERSION>

Note that code coverage is not available on Windows.

Unit tests results are available in $PWD/results/unit_tests. Coverage results are available in $PWD/results/coverage.

Sanitizers

Address Sanitizer

# Build Unit Tests with Address Sanitizer enabled (if supported)
python scripts/cmake.py <Debug|Release|RelWithDebInfo> <COMPILER_NAME> <COMPILER_VERSION> --address_sanitizer --unit_tests

# Run Unit Tests with Address Sanitizer
python scripts/tools/run_sanitizer.py --address_sanitizer install/unit_tests/teiacare_inference_client_unit_tests

Note that Address Sanitizer is supported only on Linux.

Thread Sanitizer

# Build Unit Tests with Thread Sanitizer enabled (if supported)
python scripts/cmake.py <Debug|Release|RelWithDebInfo> <COMPILER_NAME> <COMPILER_VERSION> --thread_sanitizer --unit_tests

# Run Unit Tests with Thread Sanitizer
python scripts/tools/run_sanitizer.py --thread_sanitizer install/unit_tests/teiacare_inference_client_unit_tests

Note that Thread Sanitizer is supported only on Linux.

Benchmarks

# Build Benkmarks
python scripts/cmake.py <Debug|Release|RelWithDebInfo> <COMPILER_NAME> <COMPILER_VERSION> --benchmarks --warnings

# Run Benchmarks
python scripts/tools/run_benchmarks.py <COMPILER_NAME> <COMPILER_VERSION>

Benchmarks are installed in $PWD/install/benchmarks.

Code Formatting

• clang-format

clang-format can be installed via pip using the provided scripts/requirements.txt

python scripts/tools/run_clang_format.py -r -i inference_client

Code Analysis

• clang-tidy

clang-tidy can be installed via pip using the provided scripts/requirements.txt

python scripts/tools/run_clang_tidy.py -header-filter=.* inference_client

• cppcheck

First install and setup cppcheck from your OS package manager.

# Linux
sudo apt install cppcheck

# Windows
winget install cppcheck

Then run CppCheck using the provided python script:

python scripts/tools/run_cppcheck.py <Debug|Release|RelWithDebInfo>

Generate Documentation

First install and setup Doxygen from your OS package manager.

# Linux
apt-get install doxygen graphviz

# Windows
winget install doxygen

Then run Doxygen using the provided python script:

python scripts/tools/run_doxygen.py

Documentation is now installed in $PWD/docs.

Conan Package

Local Install

Create, test and install local package.

Notes:

1. The install directory path must be a valid Conan cache (i.e. ".conan" folder) located in the current directory. So, in order to install the package in a desired repository folder, it is required to run this script from the repository folder directly.
2. The Conan package tests are automatically run during package creation. The directory test_package contains a test project that is built to validate the proper package creation.

# Create the Conan package locally
python scripts/conan/create.py <Debug|Release|RelWithDebInfo> <COMPILER_NAME> <COMPILER_VERSION>

# Build and install the test package executable
python test_package/build.py <Debug|Release|RelWithDebInfo> <COMPILER_NAME> <COMPILER_VERSION>

# Run the test package executable
$PWD/install/test_package/teiacare_inference_client_test_package

Artifactory Upload

In order to upload a Conan package to TeiaCare Artifactory server it is required to setup you local Conan client once with the following commands:

# Add TeiaCare Artifactory remote to local Conan client
conan remote add teiacare $(artifactory.url)/teiacare

# Authenticate with Artifactory credentials
conan user $(artifactory.username) -p $(artifactory.password) -r teiacare

Now it is possible to create and upload a Conan package with the following commands:

# Create the Conan package locally
python scripts/conan/create.py <Debug|Release|RelWithDebInfo> <COMPILER_NAME> <COMPILER_VERSION>

# Upload the package to Artifactory on the teicare remote
python scripts/conan/upload.py teiacare teiacare_inference_client

Contributing

In order to contribute to TeiaCareInferenceClient, please follow our contribution guidelines.

License

This project is licensed under the Apache License, Version 2.0. Copyright © 2024 TeiaCare