searxng_extra/update/

[source]

Scripts to update static data in git://searx/data/

update_ahmia_blacklist.py

[source]

This script saves Ahmia’s blacklist for onion sites.

Output file: git://searx/data/ahmia_blacklist.txt (CI Update data …).

update_currencies.py

[source]

Fetch currencies from git://searx/engines/wikidata.py engine.

Output file: git://searx/data/currencies.json (CI Update data …).

update_engine_descriptions.py

[source]

Fetch website description from websites and from git://searx/engines/wikidata.py engine.

Output file: git://searx/data/engine_descriptions.json.

searxng_extra.update.update_engine_descriptions.get_output()

From descriptions[engine][language] = [description, source] To

• output[language][engine] = description_and_source

• description_and_source can be:

  • [description, source]

  • description (if source = “wikipedia”)

  • [f”engine:lang”, “ref”] (reference to another existing description)

update_external_bangs.py

[source]

Update git://searx/data/external_bangs.json using the duckduckgo bangs from BANGS_URL.

• CI Update data …

searxng_extra.update.update_external_bangs.BANGS_URL = 'https://duckduckgo.com/bang.js'

JSON file which contains the bangs.

searxng_extra.update.update_external_bangs.merge_when_no_leaf(node)

Minimize the number of nodes

A -> B -> C

• B is child of A

• C is child of B

If there are no C equals to <LEAF_KEY>, then each C are merged into A. For example (5 nodes):

d -> d -> g -> <LEAF_KEY> (ddg)
  -> i -> g -> <LEAF_KEY> (dig)

becomes (3 nodes):

d -> dg -> <LEAF_KEY>
  -> ig -> <LEAF_KEY>

update_firefox_version.py

[source]

Fetch firefox useragent signatures

Output file: git://searx/data/useragents.json (CI Update data …).

update_engine_traits.py

[source]

Update searx.enginelib.traits.EngineTraitsMap and git://searx/languages.py

searx.enginelib.traits.EngineTraitsMap.ENGINE_TRAITS_FILE:

Persistence of engines traits, fetched from the engines.

git://searx/languages.py

Is generated from intersecting each engine’s supported traits.

The script git://searxng_extra/update/update_engine_traits.py is called in the CI Update data …

searxng_extra.update.update_engine_traits.fetch_traits_map()

Fetches supported languages for each engine and writes json file with those.

searxng_extra.update.update_engine_traits.filter_locales(traits_map: EngineTraitsMap)

Filter language & region tags by a threshold.

class searxng_extra.update.update_engine_traits.UnicodeEscape

Escape unicode string in pprint.pformat

searxng_extra.update.update_engine_traits.get_unicode_flag(locale: Locale)

Determine a unicode flag (emoji) that fits to the locale

update_osm_keys_tags.py

[source]

Fetch OSM keys and tags.

To get the i18n names, the scripts uses Wikidata Query Service instead of for example OSM tags API (side note: the actual change log from map.atownsend.org.uk might be useful to normalize OSM tags).

Output file: git://searx/data/osm_keys_tags (CI Update data …).

SPARQL_TAGS_REQUEST :

Wikidata SPARQL query that returns type-categories and types. The returned tag is Tag:{category}={type} (see get_tags()). Example:

• https://taginfo.openstreetmap.org/tags/building=house#overview

• https://wiki.openstreetmap.org/wiki/Tag:building%3Dhouse at the bottom of the infobox (right side), there is a link to wikidata: https://www.wikidata.org/wiki/Q3947 see property “OpenStreetMap tag or key” (P1282)

• https://wiki.openstreetmap.org/wiki/Tag%3Abuilding%3Dbungalow https://www.wikidata.org/wiki/Q850107

SPARQL_KEYS_REQUEST :

Wikidata SPARQL query that returns keys. Example with “payment”:

• https://wiki.openstreetmap.org/wiki/Key%3Apayment at the bottom of infobox (right side), there is a link to wikidata: https://www.wikidata.org/wiki/Q1148747 link made using the “OpenStreetMap tag or key” property (P1282) to be confirm: there is a one wiki page per key ?

• https://taginfo.openstreetmap.org/keys/payment#values

• https://taginfo.openstreetmap.org/keys/payment:cash#values

rdfs:label get all the labels without language selection (as opposed to SERVICE wikibase:label).

update_pygments.py

[source]

Update pygments style

Call this script after each upgrade of pygments

class searxng_extra.update.update_pygments.Formatter(**options)

update_locales.py

[source]

Update locale names in git://searx/data/locales.json used by Locales

• searx.locales.RTL_LOCALES

• searx.locales.LOCALE_NAMES

searxng_extra.update.update_locales.get_locale_descr(locale: Locale, tr_locale)

Get locale name e.g. ‘Français - fr’ or ‘Português (Brasil) - pt-BR’

Parameters:

• locale – instance of Locale

• tr_locale – name e.g. ‘fr’ or ‘pt_BR’ (delimiter is underscore)

update_wikidata_units.py

[source]

Fetch units from git://searx/engines/wikidata.py engine.

Output file: git://searx/data/wikidata_units.json (CI Update data …).