Skip to content

correctly evaluate the score_direction argument#10

Open
alecsandrei wants to merge 1 commit into
annahedstroem:mainfrom
alecsandrei:score-direction-fix
Open

correctly evaluate the score_direction argument#10
alecsandrei wants to merge 1 commit into
annahedstroem:mainfrom
alecsandrei:score-direction-fix

Conversation

@alecsandrei

Copy link
Copy Markdown

Description

  • The score_direction argument was not correctly propagated for estimators with score_direction == "higher". Since if score_direction was always true, the score for score_direction == "lower" estimators was correctly computed.

Implemented changes

  • Fixed the comparison in meta_evaluation.py, with an additional check that throws a ValueError on invalid values.
  • Updated the type of Estimator in configs_utils.py;
  • Fixed the tests, which passed estimators[...]["init"] as score_direction.

Minimum acceptance criteria

  • I noticed in config.py that some estimators have different score_direction when compared with Quantus. For example, Pixel-Flipping is "higher" in config.py but "lower" in Quantus, with the same situation occurring for Infidelity. Under the old bug they were accidentally correct.
  • @annahedstroem

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant