Is this project abandoned?

[johann-petrak]

It seems while there are many issues and many active users and even pull requests fixing issues, there has been no activity on this project for some time.

Is this project abandoned? Could the original authors maybe make a statement what the best way forward could be from their POV ?

[simon-clematide]

I also think wikiextractor is an important tool for flexible wikipedia processing. It definitely should be maintained.

[bdog18]

Hopefully it will be able to be maintained or at least a good fork can be maintained. I have been able to get past a couple of issues by using a different fork, but considering this is the original it would be great for it to work as intended.

[FadedSocks]

❌ What Went Wrong?
The error:

lua
Copy
Edit
ValueError: cannot find context for 'fork'
…is because the wikiextractor is trying to use the 'fork' multiprocessing method — which doesn’t exist on Windows (only on Linux/macOS). Windows uses 'spawn' instead.

✅ Fix It With This Simple Change:
We just need to tell wikiextractor to run in single process mode, or manually choose a Windows-friendly mode.

✅ Option 1: Run it in Single-Threaded Mode (no multiprocessing)
bash
Copy
Edit
wikiextractor --json --no-templates --processes 1 --output extracted enwiki-latest-pages-articles.xml.bz2
This avoids any multiprocessing and just runs it all through one process (a bit slower, but guaranteed to work).

✅ Option 2: Use Python to Call the Extractor (Recommended for Control)
If you're cool with running a quick Python script instead of the CLI, here's how:

Open a new .py file or open a terminal and paste this:

python
Copy
Edit
from wikiextractor.WikiExtractor import main
import sys

sys.argv = [
'WikiExtractor.py',
'--json',
'--no-templates',
'--output', 'extracted',
'enwiki-latest-pages-articles.xml.bz2'
]

main()
Save it as run_extractor.py

Run it like this:

bash
Copy
Edit
py -3.10 run_extractor.py

Inside your extracted/ folder:

Tons of .json or .txt files

Each contains articles in clean, readable format

Perfect for indexing, chunking, and embedding into an LLM search engine. I'm trying to use ai to help me get around this issue so I can train a local model on the wiki dump. I can't change to "spawn" without errors. Tried a direct patch and broke the ai haha

[weezymatt]

@bdog18 good suggestion. Is there a current fork that we may be able to maintain instead?

[karbalaie]

It seems Guiseppe Attardi has not contributed anything publicly during last year. I am currently working on a fix for python 3.13.

[simon-clematide]

[like] Simon Clematide reacted to your message:

…

________________________________ From: Ramin Karbalaie ***@***.***> Sent: Friday, September 19, 2025 11:13:48 AM To: attardi/wikiextractor ***@***.***> Cc: Simon Clematide ***@***.***>; Comment ***@***.***> Subject: Re: [attardi/wikiextractor] Is this project abandoned? (Issue #335) [https://avatars.githubusercontent.com/u/78555831?s=20&v=4]karbalaie left a comment (WikiExtractor/wikiextractor#335)<#335 (comment)> It seems Guiseppe Attardi has not contributed anything publicly during last year. I am currently working on a fix for python 3.13. — Reply to this email directly, view it on GitHub<#335 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAXGPEEEPUG5KWG6XI64OGL3TPQOZAVCNFSM6AAAAABX4RI4C2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGMJRG44TANJZGI>. You are receiving this because you commented.Message ID: ***@***.***>

[karbalaie]

Here it is: #348
I emailed Giuseppe and hope he responds. I think it would be better to keep this project alive with active maintainers rather than relying on forks that may not get the same recognition. Let's see if he reacts.

[AngledLuffa]

Hey Everyone,

I have some good news on the question of maintenance. My PI at Stanford NLP @manning and myself discussed things with Prof. Attardi @attardi, and it is true that he does not have much time for this project these days. However, both of us are now officially on the project both here and at PyPI.

By brief word of introduction, I've been maintaining Stanford NLP's CoreNLP and Stanza libraries for a few years now. Stanza in particular makes extensive use of WikiExtractor, and I've been relying on my own local edits to keep things going. My goal is to make a new release shortly which will fix at least some of the issues people have raised on this github.

I have merged a couple of the smaller PRs which looked like good ideas, especially the regex fix. So that should now be available on the master branch without needing to go to a fork.

I also have a bugfix for an exponential template expansion (found it in the Urdu Wikipedia, not sure if it shows up elsewhere) and I will merge that shortly as well.

I know that there are various problems running this on Windows which need fixing, and that will be the next priority fix after getting the current fixes out the door and onto PyPI.

And to answer the question people occasionally ask: a Chinese cucumber which I once grew on my balcony.