This is very mundane -- how to install (and run bazel..).

[rozim]

For the life of me I can't find any installation instructions nor by thrashing around can I figure out the right bazel invocation.

I'm guessing the instructions would be something like:

1. install bazel
2. git clone https://github.com/google/riegeli.git
3. cd riegeli
4. ./configure
5. bazel build ???

I was able to get bazel build //riegeli/base:status to work, but bazel build //riegeli/... is unhappy:
`ERROR: /Users/dave/Projects/riegeli/riegeli/tensorflow/io/BUILD:27:11: no such package 'third_party/tensorflow/core/platform':
BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.

• /Users/dave/Projects/riegeli/third_party/tensorflow/core/platform and referenced by '//riegeli/tensorflow/io:file_writer'`

I'm on a Macbook Air, the M1, and its tensorflow is a bit funky, which may be another issue
as I had to fix the output of ./configure since when you import tensorflow you get some
debug output:

 python -c "import tensorflow"
Init Plugin
Init Graph Optimizer
Init Kernel

My goal is to get a c++ and python library built.

Basic info:
uname -a: Darwin daves-air.lan 20.6.0 Darwin Kernel Version 20.6.0: Wed Jun 23 00:26:27 PDT 2021; root:xnu-7195.141.2~5/RELEASE_ARM64_T81
01 arm64

Python 3.9.4

TF: 2.5.0

thx,
Dave

[PolarBear292208]

I think the two build commands are:

Python:
bazel build python/riegeli:riegeli

C++:
bazel build riegeli/records:all

but the C++ build seems to be failing at the moment.

[QrczakMK]

All known build failures are fixed now. If it still does not build, please show error messages.

[datumest]

Currently working (as of 2023-04-15) on Linux is something like the following:

 mkdir -p ~/src/riegeli_demo
 cd !$
 python -m venv venv
 . ./venv/bin/activate
 # or for M1 macs, tensorflow-macos==v2.11.1, but I can't test this right now
 python -m pip install tensorflow==v2.11.1
 mkdir third_party
 cd third_party/
 git clone https://github.com/google/riegeli
 cd riegeli/
 ./configure
 # Bazel's cache is directory-based, and may retain old generated buildrules
 # If you've previously tried compiling against tf>=v2.12, currently broken, do this
 bazel clean --expunge
 bazel build //python:build_pip_package
 # NOTE: Currently broken for `bazel run //${THIS_TARGET} -- ${FLAGS}`
 python/build_pip_package.sh --bdist --dest $(realpath ~/src/riegeli_demo/riegeli)
 cd ~/src/riegeli_demo/
 python -m pip install riegeli/riegeli-*
 # This should run without errors
 python -c 'import riegeli'

[QrczakMK]

Riegeli is now available as Bzlmod in bazelbuild/bazel-central-registry#2186 — except for TensorFlow bindings, because TensorFlow does not support Bzlmod yet.

[DaddyWesker]

Using datumest instruction, I'm getting this error:

Starting local Bazel server and connecting to it...
WARNING: ignoring LD_PRELOAD in environment.
WARNING: For repository 'rules_proto', the root module requires module version rules_proto@4.0.0, but got rules_proto@5.3.0-21.7 in the resolved dependency graph.
ERROR: no such package '@@[unknown repo 'local_config_tf' requested from @@]//': The repository '@@[unknown repo 'local_config_tf' requested from @@]' could not be resolved: No repository visible as '@local_config_tf' from main repository
ERROR: /home/daddywesker/riegeli/python/riegeli/tensorflow/BUILD:16:10: no such package '@@[unknown repo 'local_config_tf' requested from @@]//': The repository '@@[unknown repo 'local_config_tf' requested from @@]' could not be resolved: No repository visible as '@local_config_tf' from main repository and referenced by '//python/riegeli/tensorflow:ops/_riegeli_dataset_ops.so'
ERROR: Analysis of target '//python:build_pip_package' failed; build aborted: Analysis failed
INFO: Elapsed time: 3.122s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
FAILED: 
    Fetching repository @@bazel_tools~cc_configure_extension~local_config_cc; starting
    Fetching repository @@rules_python~~python~python_3_12_x86_64-unknown-linux-gnu; starting
    Fetching repository @@abseil-cpp~; starting
    Fetching ..._daddywesker/da0f294da8459d47d65145758c460f63/external/abseil-cpp~; Extracting 20230802.1.tar.gz

I don't get what is wrong unfortunately.

[QrczakMK]

Unfortunately Riegeli/TensorFlow bindings are currently broken in open source since moving from the WORKSPACE model to Bzlmod in d3b2c5d, because TensorFlow does not support Bzlmod yet.

The upgrade from WORKSPACE to Bzlmod was long overdue, together with updating several dependencies (e.g. Abseil and Protobuf), which enabled code optimizations (e.g. absl::Cord support in Protobuf). The upgrade fixed other parts of the open source build, which got incompatible with newer versions of bazel in the meantime, and simplified the setup a lot (e.g. there is no ./configure step anymore) — at the cost of temporarily losing TensorFlow bindings.

The build was already incompatible with newer versions of TensorFlow. Actually all old versions it was once compatible with were already removed from TensorFlow sites. Riegeli/TensorFlow bindings were outdated and brittle, using precompiled TensorFlow, which ties other libraries (Abseil, Protobuf, Python) to exact versions used by the precompiled TensorFlow (but before that they were even more brittle when TensorFlow was built from source).

I am sorry, Riegeli/TensorFlow bindings are waiting for some brave soul with deep enough knowledge of bazel and TensorFlow to fix them.

[QrczakMK]

And if you are trying //python:build_pip_package, then this was never finished.

All other parts of open source Riegeli should work.

[DaddyWesker]

@QrczakMK thanks for the info. Too bad i guess. So there is no possible way to install riegeli as python lib right now?

[QrczakMK]

It should work if the usage is through bazel (Riegeli/Python tests work in open source). Pip is currently not supported, so I suppose it will not be convenient for non-bazel usages.

[DaddyWesker]

Got it, thanks. So, to build riegeli i need to just type bazel build inside of riegeli main folder?

[QrczakMK]

Yes. And to use Riegeli from another project, it should import Riegeli from its MODULE.bazel. (BTW, there should be a newer cut of Riegeli pushed later today to Bazel Central Registry.)

[douzebis]

Tried the following without success to build the python library on a Mac M1.
Any idea what I do wrong?

(venv) cibo:monarch fred$ python3 -m venv venv
(venv) cibo:monarch fred$ . venv/bin/activate
(venv) cibo:monarch fred$ git clone https://github.com/google/riegeli
Cloning into 'riegeli'...
remote: Enumerating objects: 23888, done.
remote: Counting objects: 100% (4765/4765), done.
remote: Compressing objects: 100% (604/604), done.
remote: Total 23888 (delta 4247), reused 4653 (delta 4160), pack-reused 19123 (from 1)
Receiving objects: 100% (23888/23888), 8.89 MiB | 1.76 MiB/s, done.
Resolving deltas: 100% (20421/20421), done.
(venv) cibo:monarch fred$ cd riegeli/python/riegeli
(venv) cibo:riegeli fred$ bazel build //python/riegeli:riegeli --verbose_failures
Starting local Bazel server and connecting to it...
WARNING: For repository 'com_google_absl', the root module requires module version abseil-cpp@20230802.1, but got abseil-cpp@20240116.1 in the resolved dependency graph.
INFO: Analyzed target //python/riegeli:riegeli (189 packages loaded, 6068 targets configured).
INFO: From Linking external/highwayhash~/libinstruction_sets.a:
warning: /Applications/[Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool](http://xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool): archive library: bazel-out/darwin_arm64-fastbuild/bin/external/highwayhash~/libinstruction_sets.a the table of contents is empty (no object file members in the library define global symbols)
ERROR: /private/var/tmp/_bazel_fred/18f7c7905d393625ce81a4e80b6567f9/external/protobuf~/python/BUILD.bazel:17:14: Linking external/protobuf~/python/google/protobuf/internal/_api_implementation.so failed: (Exit 1): cc_wrapper.sh failed: error executing CppLink command (from target @@protobuf~//python:google/protobuf/internal/_api_implementation.so) 
  (cd /private/var/tmp/_bazel_fred/18f7c7905d393625ce81a4e80b6567f9/sandbox/darwin-sandbox/170/execroot/_main && \
  exec env - \
    PATH='/Users/fred/code/monarch/venv/bin:/Users/fred/google-cloud-sdk/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/opt/homebrew/opt/openssl@3/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Library/TeX/texbin:/Applications/VMware [Fusion.app/Contents/Public](http://fusion.app/Contents/Public)' \
    PWD=/proc/self/cwd \
    ZERO_AR_DATE=1 \
  external/bazel_tools~cc_configure_extension~local_config_cc/cc_wrapper.sh @bazel-out/darwin_arm64-fastbuild/bin/external/protobuf~/python/google/protobuf/internal/_api_implementation.so-2.params)
# Configuration: 0d5ebff10b22bfda3f3ae92bd192687fc16f9a19af7749ad3686c9e0dd13d0b1
# Execution platform: @@platforms//host:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
ld: Undefined symbols:
  _PyModule_AddIntConstant, referenced from:
      _PyInit__api_implementation in api_implementation.o
  _PyModule_Create2, referenced from:
      _PyInit__api_implementation in api_implementation.o
  __Py_Dealloc, referenced from:
      _PyInit__api_implementation in api_implementation.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target //python/riegeli:riegeli failed to build
INFO: Elapsed time: 12.068s, Critical Path: 1.21s
INFO: 1384 processes: 1220 internal, 164 darwin-sandbox.
ERROR: Build did NOT complete successfully

It seems that Bazel does not fetch the correct python libraries and cannot resolve the symbols _PyModule_AddIntConstant etc.

In my setyup, it appears that these symbols are present in /opt/homebrew/opt/python@3.13//Frameworks/Python.framework/Versions/3.13/lib/python3.13/config-3.13-darwin/libpython3.13.dylib