[Feature Request]: panel detection / reading-order prediction

[R2008KK]

Quite recently, several solid datasets and models for manga panel detection have become available. So I think it's finally possible to start implementing the reading-order prediction idea that had been on hold due to the lack of the necessary tools. I'd be extremely grateful if someone smarter than me decided to take it on :)

https://huggingface.co/mayocream/koharu-layout-rfdetr-seg-2xl-1152
https://huggingface.co/datasets/mayocream/manga109-segmentation

[VoxelCubes]

That sounds interesting. I'll agree that the current heuristic is basic.
I'm not adept nor interested in packaging these models for python to make them usable, but should someone skilled enough to pull it off provide a package, I can take a closer look.

It's not entirely clear how reliable building a reading order is based on this, but if panels are segmented individually, then bubbles could be sorted using the original heuristic, but now with greater accuracy in respect to the page as a whole. But the panels themselves still need to be sorted somehow. These models give a segmentation, no reading order, so it's still a bit of a hack.

Thank you very much for bringing this up, it's pretty cool!

[R2008KK]

I'm not entirely sure what you mean by a Python package. Wouldn't the load_model.py file from the original repository for this model be sufficient?

[VoxelCubes]

Oh right, the first one is the model, the second a dataset, got the two mixed up it seems, while looking for a python entry point. Good catch!
I guess I can give it a try and see if the output is usable.
Some time next weekend perhaps. Feel free to give it a try as well and let me know if you have a better idea for how to derive a bubble order than what I outlined before.

[R2008KK]

Yes, I'm already testing this model.

So far, I've noticed that it handles tilted panels extremely well, which used to be one of weak points. However, unlike the mask, the bbox is strictly non-skewed, but this is apparently a problem with the dataset.

However, the main issue is still objects protruding beyond the panel boundaries.

That said, removing such objects does improve the results somewhat.

[VoxelCubes]

Hmm, it's false positive and true positive confidence values are pretty similar for more difficult pieces of text and onomatopoeia, making it hard to set a good threshold. For OCR, I guess it's easier to remove nonsense than to add missing boxes.
It is notably better at picking up text outside of bubbles than I think comic text detector would do on this sample, but for cleaning stuff that was cleaned erroneously is much harder to notice than stuff that still needs to be cleaned, so there it'd be better to err on the conservative side. So I'm not sure if it'd make sense to use this for cleaning, since it's onomatopoeia confidence levels are all very low.

To simply determine the reading order more accurately, the panel information could indeed prove useful. Removing problematic objects isn't an option when automating this, so we'd have to deal with bubbles floating outside of panels occasionally. Some kind of algorithm would be able to deal with the bubbles on the second page then, to get them into the correct order. Not really sure how that would work yet.
So for OCR purposes this could optionally be layered on top of the regular comic text detector to add in better onomatopoeia detection and somehow provide a more accurate algorithm for reading order?
What do you think?