Evaluation of Multilingual Ability to Use Spatial Deictic Expressions in Vision-Language Models

This repository contains code for the paper "Evaluation of Multilingual Ability to Use Spatial Deictic Expressions in Vision-Language Models" (link).

Setup

$ uv sync

Dataset generation

$ docker-compose up -d --build
$ docker-compose exec lvlm-dem-container bash
$ uv run --extra blender-env ./src/lvlm_demonstratives/blender/main.py

Evaluation

When using Gemma 3, set the required environment variables in .env.

$ uv run --env-file .env python src/experiment.py $TARGET_LANG $TARGET_MODEL --result $RESULT_DIR
$ uv run src/analysis.py --target path/to/result/gemma4b_{}_id.parquet

Contact

For any questions, please contact nglhdf@is.s.u-tokyo.ac.jp.

Bibtex

@inproceedings{watanabe-etal-2026-evaluation,
    title = "Evaluation of Multilingual Ability to Use Spatial Deictic Expressions in Vision-Language Models",
    author = "Watanabe, Kaito  and
      Yamamoto, Taisei  and
      Doi, Tomoki  and
      Yanaka, Hitomi",
    editor = "T.Y.S.S., Santosh  and
      Rodriguez, Juan Diego  and
      de Gibert, Ona",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 4: Student Research Workshop)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-srw.106/",
    doi = "10.18653/v1/2026.acl-srw.106",
    pages = "1203--1211",
    ISBN = "979-8-89176-393-7",
}