diff --git a/README.md b/README.md index 2de2fa9e..c349e365 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,28 @@ # WikiExtractor [WikiExtractor.py](http://medialab.di.unipi.it/wiki/Wikipedia_Extractor) is a Python script that extracts and cleans text from a [Wikipedia database dump](http://download.wikimedia.org/). -The tool is written in Python and requires Python 2.7 or Python 3.3+ but no additional library. +The tool is written in Python and requires Python 2.7 or Python 3.3+ but no additional library. Python 2 may not work properly any longer, testing may be needed. For further information, see the [project Home Page](http://medialab.di.unipi.it/wiki/Wikipedia_Extractor) or the [Wiki](https://github.com/attardi/wikiextractor/wiki). # Wikipedia Cirrus Extractor `cirrus-extractor.py` is a version of the script that performs extraction from a Wikipedia Cirrus dump. -Cirrus dumps contain text with already expanded templates. +Cirrus dumps contain text with already expanded templates. The Cirrus extractor does not suffer fron somewhat inadequate template expansion. Until WikiExtractors template expansion has been fixed this may be used instead. Although some templates, such as the one for stub articles, is not useful expanded. + +Examples:
+json output: python3 cirrus-extract.py -o wiki/test wiki/wiki-20191104-cirrussearch-content.json.gz
+text output: python3 cirrus-extract.py -o wiki/test -t wiki/wiki-20191104-cirrussearch-content.json.gz + +Text output is without titles, etc. It contains only the article texts separated with empty lines. + +Some additional switches are:
+--raw : basically no cleaning.
+--sentences : basic sentence based cleaning, based on dot and space, producing at least two sentences ending with a dot - but can be tricked by dots in names, etc. + +If you want, or do not want, every article in a separate file
+Change line 53 accordingly. Note that if you want something else than ./A/ABC/abc... as directory structure you need to change in the code. I have commented where (lines 123-127). Please, also look at line 281 for file name variations.
+Example: python3 cirrus-extract.py -o wiki/test -t --sentences wiki/wiki-20191216-cirrussearch-content.json.gz Cirrus dumps are available at: [cirrussearch](http://dumps.wikimedia.org/other/cirrussearch/). @@ -24,25 +38,65 @@ In order to speed up processing: ## Installation -The script may be invoked directly, however it can be installed by doing: - - (sudo) python setup.py install +Currently no installation. The script may be invoked directly. ## Usage The script is invoked with a Wikipedia dump file as an argument. The output is stored in several files of similar size in a given directory. Each file will contains several documents in this [document format](http://medialab.di.unipi.it/wiki/Document_Format). - usage: WikiExtractor.py [-h] [-o OUTPUT] [-b n[KMG]] [-c] [--json] [--html] - [-l] [-s] [--lists] [-ns ns1,ns2] - [--templates TEMPLATES] [--no-templates] [-r] - [--min_text_length MIN_TEXT_LENGTH] - [--filter_category path_of_categories_file] - [--filter_disambig_pages] [-it abbr,b,big] - [-de gallery,timeline,noinclude] [--keep_tables] - [--processes PROCESSES] [-q] [--debug] [-a] [-v] - [--log_file] - input +usage: WikiExtractor.py
+ [-h] [-o OUTPUT] [-b n[KMG]] [-c] [--json] [--html]
+ [-l] [-s] [--headersfooters] [--noLineAfterHeader]
+ [-no-title] [--squeeze_blank] [--for-bert]
+ [--remove-special-tokens] [--remove-html-tags]
+ [--point-separated]
+ [--restrict_pages_to RESTRICT_PAGES_TO]
+ [--max_articles MAX_ARTICLES] [--verbose] [--lists]
+ [-ns ns1,ns2] [--templates TEMPLATES] [--no-templates]
+ [-r] [--min_text_length MIN_TEXT_LENGTH]
+ [--filter_disambig_pages] [-it abbr,b,big]
+ [-de gallery,timeline,noinclude] [--keep_tables]
+ [--processes PROCESSES] [-q] [--debug] [-a]
+ [--log_file LOG_FILE] [-v]
+ [--filter_category FILTER_CATEGORY]
+ input + +## Examples (tested for "correct" output) +Debug and testing (short and fast): +python3 WikiExtractor.py -o wiki/test --templates templat.txt --max_articles 10 --verbose wiki/wiki-20191101-pages-articles.xml
+Debug and testing (more info on screen and a log): python3 WikiExtractor.py -o wiki/test --templates templat.txt --max_articles 10 --verbose --debug --log_file log.txt wiki/wiki-20191101-pages-articles.xml + +JSON (most extracted information): +python3 WikiExtractor.py -o wiki/test --filter_disambig_pages --templates templat.txt --titlefree --json --min_text_length 100 wiki/wiki-20191101-pages-articles.xml
+python3 WikiExtractor.py -o wiki/test --filter_disambig_pages --templates templat.txt --json --for-bert --min_text_length 100 wiki/wiki-20191101-pages-articles.xml + +Text only with "extra cleaning" (change --min_text_length to suit your use cases): +python3 WikiExtractor.py -o wiki/test --filter_disambig_pages --no_templates --remove-html-tags --remove-special-tokens --min_text_length 100 wiki/wiki-20191101-pages-articles.xml + +Other combinations: +python3 WikiExtractor.py -o wiki/test --headersfooters --titlefree --squeeze-blank wiki/wiki-20191101-pages-articles.xml
+python3 WikiExtractor.py -o wiki/test --titlefree --squeeze-blank wiki/wiki-20191101-pages-articles.xml
+python3 WikiExtractor.py -o wiki/test --noLineAfterHeader --squeeze-blank wiki/wiki-20191101-pages-articles.xml
+python3 WikiExtractor.py -o wiki/test --for-bert wiki/wiki-20191101-pages-articles.xml
+python3 WikiExtractor.py -o wiki/test --filter_disambig_pages --no_templates --for-bert --min_text_length 100 wiki/wiki-20191101-pages-articles.xml
+python3 WikiExtractor.py -o wiki/test --filter_disambig_pages --templates templat.txt --titlefree --json --for-bert --min_text_length 100 wiki/wiki-20191101-pages-articles.xml
+python3 WikiExtractor.py -o wiki/test --filter_disambig_pages --templates templat.txt --squeeze-blank --titlefree --max_articles 10 --remove-html-tags --min_text_length 100 wiki/wiki-20191101-pages-articles.xml
+ +Postprocessing +After running the extractor there may be a need for cleaning the output. In linux you may use any of the following examples. Please copy all the files to a safe place first. ANY ERROR IN THE CODE WILL DESTROY YOUR TEXT. You can be sure your text will be destroyed many times before you find the right cleaning scripts.
+left trim on one file: sed -i 's/^[ ]*//g' YOURTEXT
+right trim on one file: sed -i 's/[ ]*$//g' YOURTEXT
+If you want to work many files at a time use (do NOT have any othe files in the folder or subfolders):
+left trim on all files in folder or subfolder: find wiki/* -type f -exec sed -i 's/^[ ]*//g' {} \;
+right trim on all files in folder or subfolder: find wiki/* -type f -exec sed -i 's/[ ]*$//g' {} \;
+remove a line that starts with < and ends with > on all files in folder or subfolder: find wiki/* -type f -exec sed -E -i '/^<[^<]*>$/d' {} \;
+remove a line that starts with ( and ends with ) on all files in folder or subfolder: find wiki/* -type f -exec sed -E -i '/^[(][^(]*[)]$/d' {} \;
+Search Internet for variations and how to use with other operating systems. One variation would be to remove option "-i" and write changes to new files, instead of -i[nline] - although not very useful if you do more than one cleaning operation. + +For those use cases where only on large file is needed, in linux use: cat --squeeze-blank wiki/\*/\* > wiki/wiki.txt + + Wikipedia Extractor: Extracts and cleans text from a Wikipedia database dump and stores output in a @@ -59,7 +113,7 @@ Each file will contains several documents in this [document format](http://media {"id": "", "revid": "", "url":"", "title": "", "text": "..."} - Template expansion requires preprocesssng first the whole dump and + Template expansion requires preprocessing first the whole dump and collecting template definitions. positional arguments: @@ -116,6 +170,14 @@ Each file will contains several documents in this [document format](http://media from the article text --keep_tables Preserve tables in the output article text (default=False) + --headersfooters Adds header and footer to each article + (default=False) + --noLineAfterHeader Does not add line below title. Title is directly on article. + (default=False) + --titlefree No titles on articles + (default=False) + --squeeze-blank Minimize empty lines, that is, only empty lines are before/after title. + (default=False) Special: -q, --quiet suppress reporting progress info diff --git a/WikiExtractor.py b/WikiExtractor.py index 730b3bab..4c916921 100755 --- a/WikiExtractor.py +++ b/WikiExtractor.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- # ============================================================================= -# Version: 2.75 (March 4, 2017) +# Version: 3.0 (2020-01-29) # Author: Giuseppe Attardi (attardi@di.unipi.it), University of Pisa # # Contributors: @@ -17,6 +17,12 @@ # orangain (orangain@gmail.com) # Seth Cleveland (scleveland@turnitin.com) # Bren Barn +# HjalmarrSv [Thanks to: josecannete for wikiextractorforBERT, +# drgriffis for options.restrict_pages_to, +# tutcsis for --max_articles, +# michaelsilver for html.escape, +# Xiao Ling for templates_only and dropNested ref tags +# shimabukuro for --raw and --abstract_only # # ============================================================================= # Copyright (c) 2011-2017. Giuseppe Attardi (attardi@di.unipi.it). @@ -59,16 +65,23 @@ import sys import argparse import bz2 +import gzip import codecs -import cgi +import sys +if sys.version_info < (3, 2): + from cgi import escape as html_escape +else: + from html import escape as html_escape import fileinput import logging import os.path import re # TODO use regex when it will be standard import time import json +from collections import defaultdict +import queue from io import StringIO -from multiprocessing import Queue, Process, Value, cpu_count +from multiprocessing import Event, Queue, Process, Value, cpu_count from timeit import default_timer @@ -102,7 +115,7 @@ def __eq__ (self, other): # =========================================================================== # Program version -version = '2.75' +version = '3.0' ## PARAMS #################################################################### @@ -149,12 +162,65 @@ def __eq__ (self, other): ## # Whether to preserve section titles - keepSections = True, + keepSections = False, ## # Whether to preserve lists keepLists = False, + + ## + # Whether to add an empty line after header + noLineAfterHeader = False, + + ## + # Whether to add aHeader and Footer + headersfooters = False, + + ## + # Suppress repeated empty output lines + squeeze_blank = False, + + ## + # Whether to have a title on articles + titlefree = False, + + ## + # Point separated + point_separated = False, + + ## + # List of page IDs to restrict to + restrict_pages_to = None, + + ## + ## Remove tags like '<*>' + remove_html_tags = False, + + ## + # Maximum count of articles (default 0 means unlimited) + max_articles = 0, + + ## + # More information from program + verbose = False, + + ## + # Output only abstract + abstract_only = False, + + ## + # Whether only parse out templates and no extraction + only_templates = False, + + ## + # As unprocessed as possible + raw = False, + ## + # In formatnum, dot is standard, e.g. 1.5. If decimalkomma is set, use comma as decimal separator instead, e.g. 1,5. + # Does not have effect outside parsing. Text and parsed text may differ if care is not taken. + decimalcomma = False, + ## # Whether to output HTML instead of text toHTML = False, @@ -194,13 +260,18 @@ def __eq__ (self, other): log_file = None, + # xiaoling: The original authors have "small" in this discard list + # We keep small because sometimes they are used for tenure in band membership lists + # e.g., maroon 5. + # Edit by HjalmarrSv: we do not keep small bacause it introduces to much unwanted items (that could be cleaned another way) + discardElements = [ 'gallery', 'timeline', 'noinclude', 'pre', 'table', 'tr', 'td', 'th', 'caption', 'div', 'form', 'input', 'select', 'option', 'textarea', 'ul', 'li', 'ol', 'dl', 'dt', 'dd', 'menu', 'dir', 'ref', 'references', 'img', 'imagemap', 'source', 'small', - 'sub', 'sup', 'indicator' + 'sub', 'sup', 'indicator', ',mapframe', 'maplink', 'score' ], ) @@ -210,17 +281,17 @@ def __eq__ (self, other): ## # Regex for identifying disambig pages -filter_disambig_page_pattern = re.compile("{{disambig(uation)?(\|[^}]*)?}}|__DISAMBIG__") +filter_disambig_page_pattern = re.compile(r"{{[Dd]isambig(uation)?(\|[^}]*)?}}|__DISAMBIG__") ## g_page_total = 0 g_page_articl_total=0 g_page_articl_used_total=0 # page filtering logic -- remove templates, undesired xml namespaces, and disambiguation pages -def keepPage(ns, catSet, page): +def keepPage(ns, catSet, page, title): global g_page_articl_total,g_page_total,g_page_articl_used_total g_page_total += 1 - if ns != '0': # Aritcle + if ns != '0': # Article return False # remove disambig pages if desired g_page_articl_total += 1 @@ -228,6 +299,9 @@ def keepPage(ns, catSet, page): for line in page: if filter_disambig_page_pattern.match(line): return False + # check if filtering to specific list of pages + if options.restrict_pages_to and (not title in options.restrict_pages_to): + return False if len(options.filter_category_include) > 0 and len(options.filter_category_include & catSet)==0: logging.debug("***No include " + str(catSet)) return False @@ -238,6 +312,16 @@ def keepPage(ns, catSet, page): return True +def readRestrictedPageIDSet(f): + id_set = set() + with codecs.open(f, 'r', 'utf-8') as stream: + for line in stream: + id_set.add(line.strip().replace('_', ' ')) + return id_set + + + + def get_url(uid): return "%s?curid=%s" % (options.urlbase, uid) @@ -278,10 +362,19 @@ def get_url(uid): # ------------------------------------------------------------------------------ -selfClosingTags = ('br', 'hr', 'nobr', 'ref', 'references', 'nowiki') +selfClosingTags = ('BR', 'br', 'hr', 'nobr', 'ref', 'references', 'nowiki') # BR is kept, but may not be needed placeholder_tags = {'math': 'formula', 'code': 'codice'} +tag_re = re.compile(r'<[^>]+>') + +special_token_re = re.compile(r'__\S+__') + +def remove_tags(text): + return tag_re.sub('', text) + +def remove_special_tokens(text): + return special_token_re.sub('', text) def normalizeTitle(title): """Normalize title""" @@ -578,22 +671,42 @@ def write_output(self, out, text): out_str = out_str.encode('utf-8') out.write(out_str) out.write('\n') - else: + elif options.headersfooters: if options.print_revision: header = '\n' % (self.id, self.revid, url, self.title) else: header = '\n' % (self.id, url, self.title) footer = "\n\n" - if out == sys.stdout: # option -a or -o - - header = header.encode('utf-8') out.write(header) + if options.titlefree and not options.noLineAfterHeader: + out.write('\n') for line in text: if out == sys.stdout: # option -a or -o - line = line.encode('utf-8') - out.write(line) - out.write('\n') + if options.squeeze_blank: + if line != "" and line != " ": #Remove empty lines and lines with one space that occur frequently. + out.write(line) #Maybe they should not be produced to begin with. + out.write('\n') #One space lines may come from both empty articles and empty titles. + else: + out.write(line) + out.write('\n') out.write(footer) - + if options.squeeze_blank: #For separating articles when all empty lines removed with --squeeze-blank. + out.write('\n') + else: + for line in text: + if out == sys.stdout: # option -a or -o - + line = line.encode('utf-8') + if options.squeeze_blank: + if line != "" and line != " ": + out.write(line) + out.write('\n') + else: + out.write(line) + out.write('\n') + if options.squeeze_blank: + out.write('\n') + def extract(self, out): """ :param out: a memory file. @@ -603,8 +716,11 @@ def extract(self, out): # Separate header from text with a newline. if options.toHTML: title_str = '

' + self.title + '

' + elif options.noLineAfterHeader: + title_str = self.title else: title_str = self.title + '\n' + #Above added by HjalmarrSv # https://www.mediawiki.org/wiki/Help:Magic_words colon = self.title.find(':') if colon != -1: @@ -643,12 +759,31 @@ def extract(self, out): # $dom = $this->preprocessToDom( $text, $flag ); # $text = $frame->expand( $dom ); # + if options.abstract_only: + text = text.split('==')[0] text = self.transform(text) text = self.wiki2text(text) text = compact(self.clean(text)) - # from zwChan - text = [title_str] + text - + # from zwChan ed. by HjalmarrSv + if not options.titlefree: + text = [title_str] + text #does add title in the text of html and json, even + #though title in title also. use titlefree if not wanted. + if options.remove_html_tags: + text = [remove_tags(line) for line in text] + text = [line.replace(' ', ' ') for line in text] # squeezeblank only catches single space, above produces double (or moore?) + + if options.remove_special_tokens: + text = [remove_special_tokens(line) for line in text] + text = [line.replace(' ', ' ') for line in text] # squeezeblank only catches single space, above produces double (or moore?) + + if options.point_separated: + text = list(map(lambda t: t.replace('\n', '').replace('. ', '.\n').strip(), text)) + text = list(filter(lambda t: t is not '', text)) + text = list(filter(lambda t: t is not '\n', text)) + + if len(text) == 0: + return + if sum(len(line) for line in text) < options.min_text_length: return @@ -792,13 +927,28 @@ def clean(self, text): ############################################# # Cleanup text + # This code exists because of errors in wikipedia, errors in templates, errors in template expansion, + # or because of lazy or difficult transclusion. Some cleaning is needed because of the difference between input format + # and what we want as output. Ideally this part of the code should be as short as possible. Fix as much as possible + # upstream, i.e. before this code. + # text = text.replace('\t', ' ') text = spaces.sub(' ', text) text = dots.sub('...', text) - text = re.sub(' (,:\.\)\]»)', r'\1', text) - text = re.sub('(\[\(«) ', r'\1', text) + text = re.sub(r' ([,:\.\)\]»])', r'\1', text) + text = re.sub(r'([\[\(«]) ', r'\1', text) text = re.sub(r'\n\W+?\n', '\n', text, flags=re.U) # lines with only punctuations text = text.replace(',,', ',').replace(',.', '.') + text = re.sub(r']*>', '', text) + if options.cleaned: + text = re.sub(r'[(][^)(]*[)]', '', text) # removes all '( ... )', whether empty or full of text; first level from within + text = re.sub(r'[(][^)(]*[)]', '', text) # second level. If nested three levels, the third level, outmost, will appear in text, unless cleaned, as below. This would better be a loop. + text = re.sub(r'[(]', '', text) # removes unbalanced '(' or if deeply nested + text = re.sub(r'[)]', '', text) # removes unbalanced ')' or if deeply nested + text = re.sub(r'\s([\s,:\.\]»])', r'\1', text) + text = re.sub(r'([\[«])\s', r'\1', text) + # remove this: text = spaces.sub(' ', text) # when removing in sentence ' ' is created, remove. + # remove this: text = re.sub(r'\s\.', '.', text) # when removing at end of sentence ' .' is created, remove. if options.keep_tables: # the following regular expressions are used to remove the wikiml chartacters around table strucutures # yet keep the content. The order here is imporant so we remove certain markup like {| and then @@ -807,8 +957,14 @@ def clean(self, text): text = re.sub(r'!(?:\s)?style="[a-z]+:(?:\d+)%;[a-z]+:(?:#)?(?:[0-9a-z]+)?"', r'', text) text = text.replace('|-', '') text = text.replace('|', '') + text = re.sub(r'\n!\s.*\n', '', text, re.DOTALL) #remove all '! data-sort-type=number | Area(km²)' + if not options.cleaned: + text = re.sub(r'[(]\s*[)]', '', text) # removes all '()' and '( )' # this may have a negative effect on math articles + text = spaces.sub(' ', text) # removing text, without removing a space before or after, puts two spaces next to each other. + text = re.sub(r'\n\s*\n', '\n', text) #remove empty lines, mostly affects .json, because .txt cleaned after this function if options.toHTML: - text = cgi.escape(text) + text = html_escape(text, quote=False) + return text @@ -908,7 +1064,7 @@ def templateParams(self, parameters): # The '=' might occurr within an HTML attribute: # "<ref name=value" # but we stop at first. - m = re.match(' *([^=]*?) *?=(.*)', param, re.DOTALL) + m = re.match(r'\s*([^=]*)=(.*)', param, re.DOTALL) #m = re.match(' *([^=]*?) *?=(.*)', param, re.DOTALL) if m: # This is a named parameter. This case also handles parameter # assignments like "2=xxx", where the number of an unnamed @@ -1267,6 +1423,7 @@ def findMatchingBraces(text, ldelim=0): elif len(stack) == 1 and 0 < stack[0] < ldelim: # ambiguous {{{{{ }}} }} #yield m1.start() + stack[0], end + yield m1.start() + stack[0], end # Modified by Chao to solve Template expansion works well in an old version #197 cur = end break elif brac == '[': # [[ @@ -1337,6 +1494,9 @@ def findBalanced(text, openDelim=['[['], closeDelim=[']]']): # Only minimal support # FIXME: import Lua modules. +# https://commons.wikimedia.org/wiki/Commons:Lua/Modules, https://en.wikipedia.org/wiki/Wikipedia:Lua +# https://en.wikipedia.org/wiki/Category:Lua_metamodules +# def if_empty(*rest): """ @@ -1532,7 +1692,7 @@ def toRoman(n, romanNumeralMap): 'Roman': { 'main': roman_main }, - + # localised to Italian 'Numero romano': { 'main': roman_main } @@ -1546,7 +1706,9 @@ class MagicWords(object): """ One copy in each Extractor. - @see https://doc.wikimedia.org/mediawiki-core/master/php/MagicWord_8php_source.html + + https://doc.wikimedia.org/mediawiki-core/master/php/MagicWordFactory_8php_source.html + https://www.mediawiki.org/wiki/Manual:Magic_words """ names = [ '!', @@ -1624,6 +1786,7 @@ class MagicWords(object): 'localtimestamp', 'directionmark', 'contentlanguage', + 'pagelanguage', 'numberofadmins', 'cascadingsources', ] @@ -1637,25 +1800,29 @@ def __getitem__(self, name): def __setitem__(self, name, value): self.values[name] = value + # https://www.mediawiki.org/wiki/Help:Magic_words#Behavior_switches + # switches = ( '__NOTOC__', '__FORCETOC__', '__TOC__', - '__TOC__', + '__NOEDITSECTION__', '__NEWSECTIONLINK__', '__NONEWSECTIONLINK__', '__NOGALLERY__', '__HIDDENCAT__', + '__EXPECTUNUSEDCATEGORY__', '__NOCONTENTCONVERT__', '__NOCC__', '__NOTITLECONVERT__', '__NOTC__', - '__START__', - '__END__', + '__START__', # keep, but removed in r1695 and completely removed in r24784 + '__END__', # keep, but removed in 19213. '__INDEX__', '__NOINDEX__', '__STATICREDIRECT__', - '__DISAMBIG__' + '__DISAMBIG__', + '__NOGLOBAL__' ) @@ -1686,6 +1853,56 @@ def lcfirst(string): else: return '' +# Function for parsing 'formatnum:xxx.yyy|zzz' where zzz can be '|R', '|NOSEP': https://www.mediawiki.org/wiki/Help:Magic_words +# Takes an unformatted number (Arabic, no group separators and . as decimal separator) and outputs it in the localized digit script and formatted with decimal and decimal group separators, according to the wiki's default locale +# The |R parameter can be used to reverse the behavior, for use in mathematical situations: it's reliable and should be used only to deformat numbers which are known to be formatted exactly as formatnum formats them with the wiki's locale. +# Note: Lua function in Lua language: formatted_string = formatnum.formatNum(value, lang, prec, compact): +# Note: "Language:formatNum()" in MediaWiki's core libraries for Lua: +# Note: Templates: {{formatnum|1=value|2=lang|prec=prec|sep=compact}}: https://commons.wikimedia.org/wiki/Template:Formatnum, https://commons.wikimedia.org/wiki/Module:Formatnum +# Note: Template:Formatnum4: https://commons.wikimedia.org/wiki/Template:Formatnum4: example: {{Formatnum4|1|es}} → 1 +# Use: >>> '{:,}'.format(1234567890) -> '1,234,567,890' + return(string) # disable function until fixed, actually works this way + formatnum_reverse = False + string = string.strip() #strip whitespace before and after, if any + if re.search(r'\.', string): + has_dot = True + else: + has_dot = False + if re.search(r',', string): + has_comma = True + if has_dot and re.search(r',[^\.]*\.', string): #fix: check for locales with dot first + comma_first = True # true if there is both dot and a comma and comma comes first + else: + comma_first = False + else: + has_comma = False + if re.search(r'|R', string): #check for |R in which case number is already formated and should be unformated + formatnum_reverse = True + if not formatnum_reverse: + string = string.lstrip('0') # strip leading zeroes, if any, but not if |R, because formated number may be 00,001 which is 1 parsed {{formatnum:00001}} + newstring = string + + if formatnum_reverse and comma_first: # a limited |R function; note need for fix above + newstring = re.sub(r'[,\s]', '', newstring) # remove commas and space in number like 123,456.789 or 123 456.789, but not in 123,456 which we do not know what it is + + if options.decimalcomma and has_dot: # if there is a dot and we want to change dot to comma; note need for fix above + if comma_first: # check first if there are commas as group separators + newstring = re.sub(r',', '', newstring) # remove group separator comma(s) before changing to decimal separator comma, not caring for |R + newstring = re.sub(r'^(?P\d*)\.(?P\d*).*$','\g,\g', newstring) + + #if not options.decimalcomma: fix: add commas as group separator here, or something locale specific: this may never be implemented + + if newstring==string: #maybe not needed, but just in case something irregular comes in, at least arguments are removed + newstring = re.sub(r'^(?P[^|]*)\|.*','\g', newstring) # look for pipe (|), remove all from pipe and after + return newstring.strip() + +# function for parsing '#dateformat', '#formatdate'; #dateformat:2009-12-25; #formatdate:2009 dec 25 +# Use: >>> import datetime >>> d = datetime.datetime(2010, 7, 4, 12, 15, 58) +# >>> '{:%Y-%m-%d %H:%M:%S}'.format(d)'2010-07-04 12:15:58'def formatnum(string): +def formatdate(string): # |dmy, |mdy, |ISO 8601 + string = re.sub(r'^(?P[^|])\|(?P.*)','\g', string) # leave date "as is" for now, strip rest + return string + def fullyQualifiedTemplateTitle(templateTitle): """ @@ -1757,18 +1974,18 @@ def __call__(self, value1, value2): ROUND = Infix(lambda x, y: round(x, y)) - from math import floor, ceil, pi, e, trunc, exp, log as ln, sin, cos, tan, asin, acos, atan + def sharp_expr(extr, expr): """Tries converting a lua expr into a Python expr.""" try: expr = extr.expand(expr) - expr = re.sub('(?])=', '==', expr) # negative lookbehind - expr = re.sub('mod', '%', expr) # no \b here - expr = re.sub('\bdiv\b', '/', expr) - expr = re.sub('\bround\b', '|ROUND|', expr) + expr = re.sub(r'(?])=', '==', expr) # negative lookbehind + expr = re.sub(r'mod', '%', expr) # no \b here + expr = re.sub(r'\bdiv\b', '/', expr) + expr = re.sub(r'\bround\b', '|ROUND|', expr) return text_type(eval(expr)) except: return '%s' % expr @@ -1865,7 +2082,8 @@ def sharp_invoke(module, function, args): if functions: funct = functions.get(function) if funct: - return text_type(funct(args)) + return funct(*[args.get(str(i + 1)) for i in range(len(args))]) # Modified by Chao for fixing Template expansion works well in an old version #197 + #return text_type(funct(args)) return '' @@ -1895,6 +2113,12 @@ def sharp_invoke(module, function, args): '#titleparts': lambda *args: '', # not supported + '#dateformat': lambda extr, string, *rest: formatdate(string), #HjalmarrSv {{#dateformat:25 dec 2009|ymd}} {{#formatdate:dec 25,2009|dmy}} + + '#formatdate': lambda extr, string, *rest: formatdate(string), #HjalmarrSv {{#dateformat:2009-12-25|mdy}} {{#formatdate:2009 dec 25|ISO 8601}} {{#dateformat:25 decEmber|mdy}} + + '#tag': lambda *args: '', # not supported HjalmarrSv {{#tag:tagname|content|attribute1=value1|attribute2=value2}} + # This function is used in some pages to construct links # http://meta.wikimedia.org/wiki/Help:URL 'urlencode': lambda extr, string, *rest: quote(string.encode('utf-8')), @@ -1909,9 +2133,25 @@ def sharp_invoke(module, function, args): 'int': lambda extr, string, *rest: text_type(int(string)), -} + 'formatnum': lambda extr, string, *rest: formatnum(string), # was: 'formatnum': lambda extr, string, *rest: string, #HjalmarrSv: dot is decimal separator! + + 'gender': lambda *args: '', # not supported #HjalmarrSv: not relevant here, because "such as in "inform the user on his/her talk page", which is better made "inform the user on {{GENDER:$1|his|her|their}} talk page" + # https://www.mediawiki.org/wiki/Localisation#%E2%80%A6on_use_context_inside_sentences_via_GRAMMAR + + 'plural': lambda *args: '', # not supported #HjalmarrSv + + 'padleft': lambda *args: '', # not supported #HjalmarrSv + 'padright': lambda *args: '', # not supported #HjalmarrSv + # {{As of|2020|post=,}} {{update after|2030}} {{As of|since=y|2020|02|post=,}} {{As of|alt=Beginning in early 2020|2020|01|post=,}} + 'as of': lambda *args: '', # not supported #HjalmarrSv #this may be the wrong place in code for this https://en.wikipedia.org/wiki/Wikipedia:As_of + 'plural': lambda *args: '', # not supported #HjalmarrSv + + 'grammar': lambda *args: '', # not supported #HjalmarrSv +} + +#functionName<-title[:colon], args<-parts , extractor<-self def callParserFunction(functionName, args, extractor): """ Parser functions have similar syntax as templates, except that @@ -1920,6 +2160,7 @@ def callParserFunction(functionName, args, extractor): :param: args not yet expanded (see branching functions). https://www.mediawiki.org/wiki/Help:Extension:ParserFunctions + https://en.wikipedia.org/wiki/Module:Fun """ try: @@ -2027,11 +2268,32 @@ def define_template(title, page): # ---------------------------------------------------------------------- def dropNested(text, openDelim, closeDelim): + """Hack: Chunk the text into ref segments "..." and dropNested braces + within a segment. + This is to prevent unbalanced braces (e.g., "{{xyz | abc={{ | opq}}") from messing up + the nesting detection. Xiaoling. """ - A matching function for nested expressions, e.g. namespaces and tables. - """ + tagRE = re.compile('(<ref.*?>)(.+?)</ref>', re.IGNORECASE) openRE = re.compile(openDelim, re.IGNORECASE) closeRE = re.compile(closeDelim, re.IGNORECASE) + + start = 0 + segments = [] + m = tagRE.search(text, start) + while m: + segments.append(dropNested1(text[start:m.start()], openRE, closeRE)) + segments.append('{}{}</ref>'.format( + m.group(1), dropNested1(m.group(2), openRE, closeRE))) + start = m.end() + m = tagRE.search(text, start) + segments.append(dropNested1(text[start:], openRE, closeRE)) + return ''.join(segments) + + +def dropNested1(text, openRE, closeRE): + """ + A matching function for nested expressions, e.g. namespaces and tables. + """ # partition text in separate blocks { } { } spans = [] # pairs (s, e) for each partition nest = 0 # nesting level @@ -2141,8 +2403,25 @@ def replaceInternalLinks(text): label = inner[pipe + 1:].strip() res += text[cur:s] + makeInternalLink(title, label) + trail cur = end - return res + text[cur:] - + #return res + text[cur:] + # BUGFIX:(teffland) - 2/14/19 + # If we've run out of balanced [[ ]] but there is still + # a [[ in the text, then there was an imbalanced link + # (typically due to a typo in the ]]) and we remove everything in the + # that link by removing everything between it and two ']'s away + # then restart the process + # Empirically this fixes a lot of broken pages. + text = res + text[cur:] + broken_link = text.find('[[') + if broken_link > -1: + right = text[broken_link+2:].find(']') + if right > -1: + right += text[broken_link+right+2:].find(']') + if right > -1: + text = text[:broken_link] + text[broken_link+right+2:] + return replaceInternalLinks(text) + else: + return text # the official version is a method in class Parser, similar to this: # def replaceInternalLinks2(text): @@ -2445,16 +2724,17 @@ def makeInternalLink(title, label): EXT_LINK_URL_CLASS = r'[^][<>"\x00-\x20\x7F\s]' ANCHOR_CLASS = r'[^][\x00-\x08\x0a-\x1F]' ExtLinkBracketedRegex = re.compile( - '\[(((?i)' + '|'.join(wgUrlProtocols) + ')' + EXT_LINK_URL_CLASS + r'+)' + + #'\[(((?i)' + '|'.join(wgUrlProtocols) + ')' + EXT_LINK_URL_CLASS + r'+)' + + '\[((' + '|'.join(wgUrlProtocols) + ')' + EXT_LINK_URL_CLASS + r'+)' + r'\s*((?:' + ANCHOR_CLASS + r'|\[\[' + ANCHOR_CLASS + r'+\]\])' + r'*?)\]', - re.S | re.U) + re.I | re.S | re.U) # A simpler alternative: # ExtLinkBracketedRegex = re.compile(r'\[(.*?)\](?!])') EXT_IMAGE_REGEX = re.compile( r"""^(http://|https://)([^][<>"\x00-\x20\x7F\s]+) - /([A-Za-z0-9_.,~%\-+&;#*?!=()@\x80-\xFF]+)\.((?i)gif|png|jpg|jpeg)$""", - re.X | re.S | re.U) + /([A-Za-z0-9_.,~%\-+&;#*?!=()@\x80-\xFF]+)\.(gif|png|jpg|jpeg)$""", + re.I | re.X | re.S | re.U) def replaceExternalLinks(text): @@ -2521,9 +2801,11 @@ def makeExternalImage(url, alt=''): listOpen = {'*': '
    ', '#': '
      ', ';': '
      ', ':': '
      '} listClose = {'*': '
', '#': '', ';': '', ':': ''} -listItem = {'*': '
  • %s
    • ', '#': '
  • %s', ';': '
    %s
    ', - ':': '
    %s
    '} - +#listItem = {'*': '
  • %s
    • ', '#': '
  • %s', ';': '
    %s
    ', +# ':': '
    %s
    '} +# default to li for any list items +listItem = defaultdict(lambda: '
  • %s
    • ', {'*': '
  • %s
    • ', '#': '
  • %s', ';': '
    %s
    ', + ':': '
    %s
    '}) def compact(text): """Deal with headers, lists, empty sections, residuals of tables. @@ -2736,7 +3018,8 @@ def open(self, filename): tagRE = re.compile(r'(.*?)<(/?\w+)[^>]*?>(?:([^<]*)(<.*?>)?)?') # 1 2 3 4 -keyRE = re.compile(r'key="(\d*)"') +#keyRE = re.compile(r'key="(\d*)"') +keyRE = re.compile(r'key="([+-]?)(\d*)"') # karlstratos committed on Mar 17, 2018, Augmented key regex to catch plus/minus signs. catRE = re.compile(r'\[\[Category:([^\|]+).*\]\].*') # capture the category name [[Category:Category name|Sortkey]]" def load_templates(file, output_file=None): @@ -2820,7 +3103,7 @@ def pages_from(input): redirect = False elif tag == 'id' and not id: id = m.group(3) - elif tag == 'id' and id: + elif tag == 'id' and not revid: revid = m.group(3) elif tag == 'title': title = m.group(3) @@ -2853,6 +3136,14 @@ def pages_from(input): title = None page = [] +def try_put_until(event, q, value): + while not event.is_set(): + try: + return q.put(value, False, 1) + except queue.Full: + pass + + def process_dump(input_file, template_file, out_file, file_size, file_compress, process_count): @@ -2864,11 +3155,24 @@ def process_dump(input_file, template_file, out_file, file_size, file_compress, :param file_compress: whether to compress files with bzip. :param process_count: number of extraction processes to spawn. """ - + def hook_compressed_encoded(encoding): + def hook(filename, mode): + ext = os.path.splitext(filename)[1] + if ext == '.gz': + #import gzip #line to be removed, do not make gzip or bz2 local and conditional + return gzip.open(filename, mode, encoding=encoding) + elif ext == '.bz2': + #import bz2 #line to be removed, do not make gzip or bz2 local and conditional + return bz2.open(filename, mode, encoding=encoding) + else: + return open(filename, mode, encoding=encoding) + return hook + if input_file == '-': input = sys.stdin else: - input = fileinput.FileInput(input_file, openhook=fileinput.hook_compressed) + #input = fileinput.FileInput(input_file, openhook=fileinput.hook_compressed) + input = fileinput.FileInput(input_file, openhook=hook_compressed_encoded('utf-8')) # collect siteinfo for line in input: @@ -2883,6 +3187,8 @@ def process_dump(input_file, template_file, out_file, file_size, file_compress, # /mediawiki/siteinfo/base base = m.group(3) options.urlbase = base[:base.rfind("/")] + if options.verbose: + print("urlbase = ", options.urlbase) elif tag == 'namespace': mk = keyRE.search(line) if mk: @@ -2911,7 +3217,7 @@ def process_dump(input_file, template_file, out_file, file_size, file_compress, load_templates(file) file.close() else: - if input_file == '-': + if not options.templates_only and input_file == '-': # can't scan then reset stdin; must error w/ suggestion to specify template_file raise ValueError("to use templates with stdin dump, must supply explicit template-file") logging.info("Preprocessing '%s' to collect template definitions: this may take some time.", input_file) @@ -2921,6 +3227,12 @@ def process_dump(input_file, template_file, out_file, file_size, file_compress, template_load_elapsed = default_timer() - template_load_start logging.info("Loaded %d templates in %.1fs", len(options.templates), template_load_elapsed) + if options.templates_only: + logging.info('Templates generation only. Exiting ...') + if options.verbose: + print("Templates generation only. Exiting ...") + return + # process pages logging.info("Starting page extraction from %s.", input_file) extract_start = default_timer() @@ -2943,10 +3255,12 @@ def process_dump(input_file, template_file, out_file, file_size, file_compress, max_spool_length = 10000 spool_length = Value('i', 0, lock=False) + broken_pipe_event = Event() + # reduce job that sorts and prints output reduce = Process(target=reduce_process, args=(options, output_queue, spool_length, - out_file, file_size, file_compress)) + out_file, file_size, file_compress, broken_pipe_event)) reduce.start() # initialize jobs queue @@ -2957,56 +3271,69 @@ def process_dump(input_file, template_file, out_file, file_size, file_compress, workers = [] for i in range(worker_count): extractor = Process(target=extract_process, - args=(options, i, jobs_queue, output_queue)) + args=(options, i, jobs_queue, output_queue, broken_pipe_event)) extractor.daemon = True # only live while parent process lives extractor.start() workers.append(extractor) # Mapper process - page_num = 0 - for page_data in pages_from(input): - id, revid, title, ns, catSet, page = page_data - if keepPage(ns, catSet, page): - # slow down - delay = 0 - if spool_length.value > max_spool_length: - # reduce to 10% - while spool_length.value > max_spool_length/10: - time.sleep(10) - delay += 10 - if delay: - logging.info('Delay %ds', delay) - job = (id, revid, title, page, page_num) - jobs_queue.put(job) # goes to any available extract_process - page_num += 1 - page = None # free memory - - input.close() - - # signal termination + try: + page_num = 0 + for page_data in pages_from(input): + if broken_pipe_event.is_set(): + break + id, revid, title, ns, catSet, page = page_data + if keepPage(ns, catSet, page, title): + # slow down + delay = 0 + if spool_length.value > max_spool_length: + # reduce to 10% + while spool_length.value > max_spool_length/10: + time.sleep(10) + delay += 10 + if delay: + logging.info('Delay %ds', delay) + job = (id, revid, title, page, page_num) + # TODO if pipe is closed + try_put_until(broken_pipe_event, jobs_queue, job) # goes to any available extract_process + page_num += 1 + if options.max_articles and page_num >= options.max_articles: # + break # any need to cleanup before break? + page = None # free memory + + input.close() + + except KeyboardInterrupt: + logging.warn("Exiting due interrupt") + + # signal termination, either catch the pipe error on ctrl C or recode for _ in workers: - jobs_queue.put(None) - # wait for workers to terminate + jobs_queue.put(None) + + # wait for workers to terminate, this alone leaves all processes as zombies, above is needed + # it may still leave a couple of zombie processes even with tha above code - fix this + # https://docs.python.org/3/library/multiprocessing.html for w in workers: w.join() - # signal end of work to reduce process - output_queue.put(None) - # wait for it to finish - reduce.join() + if reduce.is_alive(): + # signal end of work to reduce process + output_queue.put(None) + # wait for reduce process to finish + reduce.join() extract_duration = default_timer() - extract_start extract_rate = page_num / extract_duration logging.info("Finished %d-process extraction of %d articles in %.1fs (%.1f art/s)", process_count, page_num, extract_duration, extract_rate) - logging.info("total of page: %d, total of articl page: %d; total of used articl page: %d" % (g_page_total, g_page_articl_total,g_page_articl_used_total)) + logging.info("total of pages: %d, total of article pages: %d; total of used article pages: %d" % (g_page_total, g_page_articl_total,g_page_articl_used_total)) # ---------------------------------------------------------------------- # Multiprocess support -def extract_process(opts, i, jobs_queue, output_queue): +def extract_process(opts, i, jobs_queue, output_queue, broken_pipe_event): """Pull tuples of raw page content, do CPU/regex-heavy fixup, push finished text :param i: process id. :param jobs_queue: where to get jobs. @@ -3021,31 +3348,41 @@ def extract_process(opts, i, jobs_queue, output_queue): out = StringIO() # memory buffer - while True: - job = jobs_queue.get() # job is (id, title, page, page_num) - if job: - id, revid, title, page, page_num = job - try: - e = Extractor(*job[:4]) # (id, revid, title, page) - page = None # free memory - e.extract(out) - text = out.getvalue() - except: - text = '' - logging.exception('Processing page: %s %s', id, title) - - output_queue.put((page_num, text)) - out.truncate(0) - out.seek(0) - else: - logging.debug('Quit extractor') - break + try: + while not broken_pipe_event.is_set(): + job = jobs_queue.get() # job is (id, title, page, page_num) + if job: + id, revid, title, page, page_num = job + try: + e = Extractor(*job[:4]) # (id, revid, title, page) + page = None # free memory + e.extract(out) + text = out.getvalue() + except Exception: + text = '' + logging.exception('Processing page: %s %s', id, title) + + try_put_until(broken_pipe_event, output_queue, (page_num, text)) + out.truncate(0) + out.seek(0) + else: + logging.debug('Quit extractor') + break + + except KeyboardInterrupt: + logging.info('Aborting worker %d', i) + output_queue.cancel_join_thread() + jobs_queue.cancel_join_thread() + + if broken_pipe_event.is_set(): + output_queue.cancel_join_thread() + out.close() report_period = 10000 # progress report period def reduce_process(opts, output_queue, spool_length, - out_file=None, file_size=0, file_compress=True): + out_file, file_size, file_compress, broken_pipe_event): """Pull finished article text, write series of files (or stdout) :param opts: global parameters. :param output_queue: text to be output. @@ -3072,33 +3409,44 @@ def reduce_process(opts, output_queue, spool_length, # FIXME: use a heap spool = {} # collected pages next_page = 0 # sequence numbering of page - while True: - if next_page in spool: - output.write(spool.pop(next_page).encode('utf-8')) - next_page += 1 - # tell mapper our load: - spool_length.value = len(spool) - # progress report - if next_page % report_period == 0: - interval_rate = report_period / (default_timer() - interval_start) - logging.info("Extracted %d articles (%.1f art/s)", - next_page, interval_rate) - interval_start = default_timer() - else: - # mapper puts None to signal finish - pair = output_queue.get() - if not pair: - break - page_num, text = pair - spool[page_num] = text - # tell mapper our load: - spool_length.value = len(spool) - # FIXME: if an extractor dies, process stalls; the other processes - # continue to produce pairs, filling up memory. - if len(spool) > 200: - logging.debug('Collected %d, waiting: %d, %d', len(spool), - next_page, next_page == page_num) - if output != sys.stdout: + try: + while True: + if next_page in spool: + try: + output.write(spool.pop(next_page).encode('utf-8')) + except BrokenPipeError: + # other side of pipe (like `head` or `grep`) is closed + # we can simply exit + broken_pipe_event.set() + break + + next_page += 1 + # tell mapper our load: + spool_length.value = len(spool) + # progress report + if next_page % report_period == 0: + interval_rate = report_period / (default_timer() - interval_start) + logging.info("Extracted %d articles (%.1f art/s)", + next_page, interval_rate) + interval_start = default_timer() + else: + # mapper puts None to signal finish + pair = output_queue.get() + if not pair: + break + page_num, text = pair + spool[page_num] = text + # tell mapper our load: + spool_length.value = len(spool) + # FIXME: if an extractor dies, process stalls; the other processes + # continue to produce pairs, filling up memory. + if len(spool) > 200: + logging.debug('Collected %d, waiting: %d, %d', len(spool), + next_page, next_page == page_num) + except KeyboardInterrupt: + pass + + if output != sys.stdout and not broken_pipe_event.is_set(): output.close() @@ -3133,13 +3481,45 @@ def main(): help="preserve links") groupP.add_argument("-s", "--sections", action="store_true", help="preserve sections") + + groupP.add_argument("--headersfooters", action="store_true", + help="adds header and footer to each article") + groupP.add_argument("--noLineAfterHeader", action="store_true", + help="does not add line below heading") + groupP.add_argument("-no-title", "--titlefree", action="store_true", + help="no titles on articles") + groupP.add_argument("--squeeze_blank", "--squeeze-blank", action="store_true", + help="suppress repeated empty output lines") + groupP.add_argument("--for-bert", action="store_true", + help="ready for bert pre-training") + groupP.add_argument("--remove-special-tokens", action="store_true", + help="to remove every special token ie: between '__*__'") + groupP.add_argument("--remove-html-tags", action="store_true", + help="to remove every html tag ie: between '<*/>'") + groupP.add_argument("--point-separated", action="store_true", + help="every line is separated") + groupP.add_argument("--restrict_pages_to", default=None, + help="List of page IDs to restrict to (one per line, case-sensitive)") + groupO.add_argument("--max_articles", type=int, default=0, + help="maximum count of articles (default 0 means unlimited)") + groupO.add_argument("--verbose", action="store_true", + help="display extended information") + groupP.add_argument("--only_templates", action="store_true", + help="only generates or loads templates file, no extraction.") + groupP.add_argument("--raw", action="store_true", + help="parse raw media wiki for debug") + groupP.add_argument("--abstract_only", action="store_true", + help="output text only abstract content") + groupP.add_argument("--decimalcomma", action="store_true", + help="use comma, instead of dot, as decimal separator") + groupP.add_argument("--lists", action="store_true", help="preserve lists") groupP.add_argument("-ns", "--namespaces", default="", metavar="ns1,ns2", help="accepted namespaces in links") groupP.add_argument("--templates", help="use or create file containing templates") - groupP.add_argument("--no_templates", action="store_false", + groupP.add_argument( "--no-templates", "--no_templates", action="store_false", help="Do not expand templates") groupP.add_argument("-r", "--revision", action="store_true", default=options.print_revision, help="Include the document revision id (default=%(default)s)") @@ -3183,11 +3563,39 @@ def main(): options.min_text_length = args.min_text_length if args.html: options.keepLinks = True + + options.noLineAfterHeader = args.noLineAfterHeader + options.headersfooters = args.headersfooters + options.titlefree = args.titlefree + options.squeeze_blank = args.squeeze_blank + options.point_separated = args.point_separated + options.remove_html_tags = args.remove_html_tags + options.remove_special_tokens = args.remove_special_tokens + options.max_articles = args.max_articles + options.verbose = args.verbose + options.templates_only = args.only_templates + options.abstract_only = args.abstract_only + options.raw = args.raw + options.decimalcomma = args.decimalcomma + options.cleaned = False options.expand_templates = args.no_templates options.filter_disambig_pages = args.filter_disambig_pages options.keep_tables = args.keep_tables + if args.restrict_pages_to: + options.restrict_pages_to = readRestrictedPageIDSet(args.restrict_pages_to) + else: #Not strictly needed since None already declared twice, + options.restrict_pages_to = None #part of imported code that may be changed. + + if args.for_bert: + options.headersfooters = False + options.titlefree = True + options.point_separated = True + options.squeeze_blank = True #Here it actually adds an empty line between articles. Otherwise it is not needed. + options.remove_html_tags = True + options.remove_special_tokens = True + try: power = 'kmg'.find(args.bytes[-1].lower()) + 1 file_size = int(args.bytes[:-1]) * 1024 ** power @@ -3210,6 +3618,8 @@ def main(): 'p', 'plaintext', 's', 'span', 'strike', 'strong', 'tt', 'u', 'var' ] + if options.verbose: + print("ignored tags = ", ignoredTags) # 'a' tag is handled separately for tag in ignoredTags: @@ -3217,6 +3627,9 @@ def main(): if args.discard_elements: options.discardElements = set(args.discard_elements.split(',')) + + if options.verbose: + print("discard elements = ", options.discardElements) FORMAT = '%(levelname)s: %(message)s' logging.basicConfig(format=FORMAT) @@ -3230,6 +3643,12 @@ def main(): if not options.keepLinks: ignoreTag('a') + + if options.raw: + file = fileinput.FileInput(input_file, openhook=fileinput.hook_compressed) + Extractor(0, 0, "raw", file).extract(open("raw.json", "w")) + return + # sharing cache of parser templates is too slow: # manager = Manager() diff --git a/cirrus-extract.py b/cirrus-extract.py index 895970da..63ca0478 100755 --- a/cirrus-extract.py +++ b/cirrus-extract.py @@ -2,8 +2,9 @@ # -*- coding: utf-8 -*- # # ============================================================================= -# Version: 1.00 (December 15, 2015) -# Author: Giuseppe Attardi (attardi@di.unipi.it), University of Pisa +# Version: 1.12 (Mars 8, 2020) +# Author(s): Giuseppe Attardi (attardi@di.unipi.it), University of Pisa +# HjalmarrSv # # ============================================================================= # Copyright (c) 2015. Giuseppe Attardi (attardi@di.unipi.it). @@ -43,9 +44,15 @@ import logging # Program version -version = '1.00' +version = '1.11' -urlbase = 'http://it.wikipedia.org/' +# Urlbase +urlbase = 'http://de.wikipedia.org/' + +# Numbered files is default output. Change to False if you want article files in "articled" directories. +numbered = True +# When article output is chosen above, there will be global differences between text_only and jason_like output. +textonly = False # ---------------------------------------------------------------------- @@ -56,27 +63,84 @@ class NextFile(object): filesPerDir = 100 - def __init__(self, path_name): + def __init__(self, path_name, title): self.path_name = path_name - self.dir_index = -1 - self.file_index = -1 + self.dir_index = -1 # for enumerated file names + self.file_index = -1 # + self.title = title # for article file names def next(self): - self.file_index = (self.file_index + 1) % NextFile.filesPerDir - if self.file_index == 0: - self.dir_index += 1 - dirname = self._dirname() - if not os.path.isdir(dirname): - os.makedirs(dirname) - return self._filepath() + if numbered: + self.file_index = (self.file_index + 1) % NextFile.filesPerDir + if self.file_index == 0: + self.dir_index += 1 + dirname = self._dirname() + if not os.path.isdir(dirname): + os.makedirs(dirname) + return self._filepath() + else: + dirname = self._dirname() + if not os.path.isdir(dirname): + os.makedirs(dirname) + return self._filepath() def _dirname(self): - char1 = self.dir_index % 26 - char2 = self.dir_index / 26 % 26 - return os.path.join(self.path_name, '%c%c' % (ord('A') + char2, ord('A') + char1)) + if numbered: + char1 = self.dir_index % 26 + char2 = self.dir_index // 26 % 26 + return os.path.join(self.path_name, '%c%c' % (ord('A') + char2, ord('A') + char1)) + else: + # Remove punctuation from title, for keeping paths from failing (both from path name and filename) + # self.title = self.title.replace('.', '') # should not be a problem in title/filename + # self.title = self.title.replace('\\', '') # not a problem in linux: + if textonly: # less replacements needed for jason-like article titles + # + self.title = self.title.replace('/', '÷') # replace '' or '÷' https://stackoverflow.com/questions/1976007/what-characters-are-forbidden-in-windows-and-linux-directory-names + title = self.title # If you want to limit the amount of symbols for directory names, clean title accordingly. + title = title.replace('.', '') # No dots in directory names. + title = title.upper() # Use caps for directory names (G/GW, or else g/gw G/Gw and G/GW are possible, and likely). + #if len(title) > 0: # Do not create directories for articles on (, [ (if any), by cleaning below. + if len(title) > 1: # Create directories for articles on (, [ (if any) by not cleaning one character articles below. + title = re.sub(r'[’"”\(\[\)\]]', "", title) # No citation marks, or (, [ in directory names (no reason, but does not look nice). + title = title.lstrip() # Remove space at beginning, since it may occur. + if len(title)==0: # This can happen if "clean" title and title is only "." or other that is removed above. + return self.path_name + else: + dirname2 = dirname3 = dirname4 = "" # initialize before use (short titles will fail otherwise) + title = re.sub(r'_.*$', "", title) # Do not add version nr, etc to directory structure (if added before). + if title=="": + #title = "____" # replace (./__/____/____[_1234]) with your choice, for the possible few articles consisting + return self.path_name # of only the few forbidden file characters if uncommenting previous line and commenting this one. + dirname1 = title[0] + if len(title)==1: + dirname2 = dirname3 = "" # note "a" not in ./aa/aa but in ./a/a (both ./a/a and ./aa/aa will exist, change + else: # with: dirname2 = dirname3 = dirname1 (also below), then "a" will be in ./aa/aa + dirname2 = title[1] # present choice for machine readability + if len(title)==2: + dirname3 = "" + else: + dirname3 = title[2] + if len(title)==3: + dirname4 = "" + else: + dirname4 = title[3] + #if not textonly: # A/AB/abc for jason-like articles. Less articles expected in this format, maybe about a million, + # because basically one directory level is inside the article that contains many articles. + # + #else: # A/ABC/abc for text only output, because of the large number of articles, tens of millions. + d1 = dirname1 # first level directory - first letter (change to your needs) + dir_path_name = os.path.join(self.path_name, d1) + d2 = dirname1 + dirname2 + dirname3 # + dirname4 # second level directory - all three first letters (could be next two letters) + d2 = re.sub(r'[ ,]$', "", d2) # remove ending space and comma, if any. + dir_path_name = os.path.join(dir_path_name, d2) # two directories deep, change if needed + return dir_path_name + def _filepath(self): - return '%s/wiki_%02d' % (self._dirname(), self.file_index) + if numbered: + return '%s/wiki_%02d' % (self._dirname(), self.file_index) + else: + return '%s/%s' % (self._dirname(), self.title) class OutputSplitter(object): """ @@ -85,7 +149,7 @@ class OutputSplitter(object): def __init__(self, nextFile, max_file_size=0, compress=True): """ - :param nextfile: a NextFile object from which to obtain filenames + :param nextFile: a NextFile object from which to obtain filenames to use. :param max_file_size: the maximum size of each file. :para compress: whether to write data with bzip compression. @@ -111,7 +175,7 @@ def open(self, filename): if self.compress: return bz2.BZ2File(filename + '.bz2', 'w') else: - return open(filename, 'w') + return open(filename, 'wb') # ---------------------------------------------------------------------- @@ -136,7 +200,7 @@ def extract(self, out): out.write('\n') out.write(footer) -def process_dump(input_file, out_file, file_size, file_compress): +def process_dump(input_file, out_file, file_size, file_compress, text_only, sentences_only, raw_only): """ :param input_file: name of the wikipedia dump file; '-' to read from stdin :param out_file: directory where to store extracted data, or '-' for stdout @@ -153,14 +217,24 @@ def process_dump(input_file, out_file, file_size, file_compress): output = sys.stdout if file_compress: logging.warn("writing to stdout, so no output compression (use external tool)") - else: - nextFile = NextFile(out_file) + elif numbered: + title = "" + nextFile = NextFile(out_file, title) output = OutputSplitter(nextFile, file_size, file_compress) # process dump - # format + # format (two lines: index and content) # {"index":{"_type":"page","_id":"3825914"}} # {"namespace":0,"title":TITLE,"timestamp":"2014-06-29T15:51:09Z","text":TEXT,...} + # or + # {"index":{"_type":"page","_id":"4274592"}} + # {"template":["Template:See also","Template:Studebaker","Template:Navbox","Template:Truck-stub","Template:Asbox","Module:Labelled list + # hatnote","Module:Hatnote","Module:Hatnote + # list","Module:Arguments","Module:Navbox","Module:Navbar","Module:Asbox","Module:Buffer"], + # "content_model":"wikitext","opening_text":"Transtar was ... 1963.","wiki":"enwiki","auxiliary_text":[ + # "This truck-related article is a stub. You can help Wikipedia by expanding it. v t e","Transtar Truck Model + # 3-E"],"language":"en","title":"Studebaker Transtar","text":"Transtar + while True: line = input.readline() if not line: @@ -169,18 +243,61 @@ def process_dump(input_file, out_file, file_size, file_compress): content = json.loads(input.readline()) type = index['index']['_type'] id = index['index']['_id'] - language = content['language'] - revision = content['version'] - if type == 'page' and content['namespace'] == 0: + try: + language = content['language'] # use try: because en errors out on some lines + except: + language = "" + try: + revision = content['version'] + except: + revision = "0" + # date from timestamp + try: + time = content['timestamp'] + date = re.sub(r'[^d]*(?P\d{4})-(?P\d{2})-(?P\d{2}).*', '\g\g\g', time) + except: + time = date = "0" + try: + ns = content['namespace'] + except: + ns = -1 + + if type == 'page' and ns == 0: title = content['title'] text = content['text'] - # drop references: - # ^ The Penguin Dictionary - text = re.sub(r' \^ .*', '', text) - url = urlbase + 'wiki?curid=' + id - header = '\n' % (id, url, title, language, revision) - page = header + title + '\n\n' + text + '\n\n' - output.write(page.encode('utf-8')) + text = re.sub(r'\s', ' ', text) # change different whitespace characters to conventional space, e.g. wikipedia contains tabs + text = re.sub(r' ( )+', ' ', text) # replace multiple consecutive spaces, if any (e.g.tab+space), with a single space + text = text.strip() # remove space in beginning and end + if not raw_only: + # drop references: + # ^ The Penguin Dictionary + text = re.sub(r' \^ .*', '', text) # only one space before caret to catch malformed tags + if sentences_only: + text = re.sub(r'[.!?] [^.!?]*[^.!?]$', '.', text) # remove incomplete last sentence of at least two, no ending dot + text = re.sub(r'^.*[^.!?]$', '', text) # now remove incomplete only sentence, no ending dot. this article will not be. + #text = re.sub(r'\. [^.]*$', '.', text) # remove incomplete last sentence, no ending dot + #text = re.sub(r'\.\s(.(?!\.\s))*[^.]$', '.', text) # remove incomplete last sentence, no dot and space found as separator and no ending dot + #text = re.sub(r'^[^.]*$', '', text) # remove incomplete sentence, even if only sentence in article, no dot at all + #text = re.sub(r'^(.(?!\.\s))*$', '', text) # remove if only one sentence in article, no dot and space found as separator + if text != "" and text != " ": # do not create empty articles + url = urlbase + 'wiki?curid=' + id + header = '\n' % (id, url, title, language, revision) + if not text_only: + page = header + title + '\n\n' + text + '\n\n' + else: + page = text + '\n\n' + if numbered: + output.write(page.encode('utf-8')) + #elif not text_only # json-like output, where every word only has one article, and this article has all variations in it + # #check if article exists, and if article exists check for id and version, if not exist then add or create + else: + title = re.sub(r'_', ' ', title) # Replace all _ since they are reserved for use below. + title = title + "_" + language + "_"+ id + "_" + str(revision) + "_" + date # remove this line if clean articles wanted (note the + nextFile = NextFile(out_file, title) # increased risk for overwriting articles). Chose if you want all revisions, + output = OutputSplitter(nextFile, file_size, file_compress) # for all your wiki runs, by adding/removing # + output.write(page.encode('utf-8')) + page = "" + # ---------------------------------------------------------------------- @@ -201,7 +318,15 @@ def main(): metavar="n[KMG]") groupO.add_argument("-c", "--compress", action="store_true", help="compress output files using bzip") - + groupO.add_argument("-t", "--text", action="store_true", + help="text only") + groupO.add_argument("-s", "--sentences", action="store_true", + help="Only at least two complete point separated sentences.") + groupO.add_argument("-r", "--raw", action="store_true", + help="No filtering.") + # groupO.add_argument("-a", "--articles", action="store_true", + # help="Output as separate articles.") + groupP = parser.add_argument_group('Processing') groupP.add_argument("-ns", "--namespaces", default="", metavar="ns1,ns2", help="accepted namespaces") @@ -214,6 +339,10 @@ def main(): help="print program version") args = parser.parse_args() + + # numbered = True + # if args.articles: + # numbered = False try: power = 'kmg'.find(args.bytes[-1].lower()) + 1 @@ -241,7 +370,8 @@ def main(): logging.error('Could not create: %s', output_path) return - process_dump(input_file, output_path, file_size, args.compress) + + process_dump(input_file, output_path, file_size, args.compress, args.text, args.sentences, args.raw) if __name__ == '__main__':