Commit Graph

1175 Commits

Author SHA1 Message Date
Allen 0c351ea364
[enh] Add Tineye reverse image search (#3040)
* [enh] Add Tineye reverse image search 

Other optional parametesr:

"&sort=crawl_date" can be appended to search_string to sort results by date.
"&domain=example.org" can be implemented to search_string to get results from just one domain.

Public instances could get relatively fast timed-out for 3600s.

* [enh] Add TIneye to settings.yml 

Check if that's the right shortcut.

* [mod] Fix checks

* [mod] Try to fix checks

* [mod] Use Four spaces for indentation

And set paging back to True

Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
2022-01-22 12:15:19 +01:00
Noémi Ványi 148090df12 Minor fixes to satisfy the linter 2022-01-21 17:59:10 +01:00
Alexandre Flament d592159cc5 [fix] startpage: workaround to use the startpage network
workaround for the issue #762
2022-01-21 17:59:10 +01:00
Markus Heiser 036d80ed20 [mod] starpage engine: add comment about Startpage's FFox add-on
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-21 17:59:10 +01:00
Markus Heiser a4bc089091 [fix] startpage engine: fetch CAPTCHA & issues related to PR-695
In case of CAPTCHA raise a SearxEngineCaptchaException and suspend for 7 days.
When get_sc_code() fails raise a SearxEngineResponseException and suspend for 7
days.

[1] https://github.com/searxng/searxng/pull/695

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-21 17:59:10 +01:00
Markus Heiser 1076d7e52e [fix] Get an actual `sc` argument from startpage's home page.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-21 17:59:10 +01:00
Markus Heiser a6184ac32c [pylint] Startpage engine
Fix remarks from pylint

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-21 17:59:10 +01:00
Markus Heiser 4750586fb0 [fix] startpage engine - avoid captcha
Startpage has introduced new anti-scraping measures that make SearXNG instances
run into captchas:

1. some arguments has been removed and a new `sc` has been added.
2. search path changed from `do/search` to `sp/search`
3. POST request is no longer needed

Closes: https://github.com/searxng/searxng/issues/692
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-21 17:59:10 +01:00
Markus Heiser 99128537a8 [fix] googel engine - "some results are invalids: invalid content"
Fix google issues listet in the `/stats?engine=google` and message::

    some results are invalids: invalid content

The log is::

    DEBUG   searx                         : result: invalid content: {'url': 'https://de.wikipedia.org/wiki/Foo', 'title': 'Foo - Wikipedia', 'content': None, 'engine': 'google'}
    WARNING searx.engines.google          : ErrorContext('searx/search/processors/abstract.py', 111, 'result_container.extend(self.engine_name, search_results)', None, 'some results are invalids: invalid content', ()) True

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-21 17:59:10 +01:00
Markus Heiser 26c92d5f50 [fix] google engine: remove adds and fix mobile_ui selector
1. Fix issue reported in comment [1]
2. Fix XPath selector for the response of google's mobile UI, reported in
   comment [2]

[1] https://github.com/searxng/searxng/pull/777#issuecomment-1015121322
[2] https://github.com/searxng/searxng/pull/777#issuecomment-1015236238

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-20 08:33:53 +01:00
Émilien Devos a2ec27696c Update XPath for Google engine 2022-01-19 23:03:36 +01:00
Noémi Ványi 179784068f
Bump pylint from 2.10.2 to 2.12.2 (#3124)
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.10.2 to 2.12.2.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.10.2...v2.12.2)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-01-15 20:23:09 +01:00
Dario Nuevo 1a18adcc16
New files engine: Prowlarr (#3118)
## What does this PR do?

Gives the user the possibility to search their own prowlarr instances.

Info: https://wiki.servarr.com/en/prowlarr
Github: https://github.com/Prowlarr/Prowlarr

## Why is this change important?

Prowlarr searchs multiple upstream search providers, thus allows to use that functionality through searx.
2022-01-15 19:18:15 +01:00
Andy Jones 3ddd0f8944
Update httpx and friends to 0.21.3 (#3121) 2022-01-15 19:16:10 +01:00
Noémi Ványi 82ac634070 make port configurable in MySQL engine
Closes #3117
2022-01-11 22:49:53 +01:00
Dario Nuevo 8f07442fb6
feature: new engine xpath_flex (#3119) 2022-01-11 22:44:19 +01:00
Finn 5dc886136b
[fix] Qwant: Remove extra q from URL (#3091)
Fixes #3090
2022-01-07 21:41:39 +01:00
israelyago 0d28fd2efe
Merge branch 'master' into onesearch-engine 2021-11-17 15:27:11 -03:00
Israel Yago Pereira f1f3ad97d9 Remove debug log from onesearch engine 2021-11-17 15:15:17 -03:00
Israel Yago Pereira 4b785677d8 Onesearch pagination 2021-11-17 15:14:43 -03:00
Israel Yago Pereira 51530bc394 Fix code style 2021-11-17 15:14:43 -03:00
Israel Yago Pereira 258c6fbd5a Onesearch engine without pagination 2021-11-17 15:14:43 -03:00
Markus Heiser 4d36aee57b [fix] engine - yahoo: rewrite and fix issues
Languages are supported by mapping the language to a domain.  If domain is not
found in :py:obj:`lang2domain` URL ``<lang>.search.yahoo.com`` is used.

Closes #3020

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-16 20:30:10 +01:00
Israel Yago Pereira a5fd30bf4d fix wrong func call 2021-11-12 13:12:50 -03:00
Noémi Ványi 3bcca43abf Fix qwant engine, only get results from categories
Closes #3014
2021-10-12 20:06:37 +02:00
Noémi Ványi a0fb8ebeaf Fix style errors in Qwant engine 2021-10-10 21:13:55 +02:00
Markus Heiser 263db54aa9 [fix] qwant engine - prevent API locale exception on lang 'all'
Has been reported in [1], error message::

    Error
        Error: searx.exceptions.SearxEngineAPIException
        Percentage: 0
        Parameters: ('API error::locale must be a string,locale must be one of
        the following values: en_gb, en_ie, en_us, en_ca, en_in, en_my, en_au,
        en_nz, cy_gb, gd_gb, de_de, de_ch, de_at, fr_fr, br_fr, fr_be, fr_ch,
        fr_ca, fr_ad, fc_ca, ec_ca, co_fr, es_es, es_ar, es_cl, es_co, es_mx,
        es_pe, es_ad, ca_es, ca_ad, ca_fr, eu_es, eu_fr, it_it, it_ch, pt_br,
        pt_pt, pt_ad, nl_be, nl_nl, pl_pl, zh_hk, zh_cn, fi_fi, bg_bg, et_ee,
        hu_hu, da_dk, nb_no, sv_se, ko_kr, th_th, cs_cz, ro_ro, el_gr',)
        File name: searx/engines/qwant.py:114
        Function: response
        Code: raise SearxEngineAPIException('API error::' + msg)

[1] https://github.com/searxng/searxng/issues/222

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-10 21:08:48 +02:00
Markus Heiser b10c1346d7 [fix] qwant engine - prevent exception on date/time value is None
Has been reported in [1], error messages::

  Error
       Error: ValueError
       Percentage: 0
       Parameters: ()
       File name: searx/engines/qwant.py:159
       Function: response
       Code: pub_date = datetime.fromtimestamp(item['date'], None)

    Error
        Error: TypeError
        Percentage: 0
        Parameters: ('an integer is required (got type NoneType)',)
        File name: searx/engines/qwant.py:196
        Function: response
       Code: pub_date = datetime.fromtimestamp(item['date'])

Fix timedelta from seconds to milliseconds [1], error message::

    Error
        Error: TypeError
        Percentage: 0
        Parameters: ('unsupported type for timedelta seconds component: NoneType',)
        File name: searx/engines/qwant.py:195
        Function: response
        Code: length = timedelta(seconds=item['duration'])

[1] https://github.com/searxng/searxng/issues/222

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-10 21:08:48 +02:00
Alexandre Flament 7aa94b7084 [mod] qwant engine: fix typos / minor change
minor modification of commit 628b5703f3aeeed117772696f83efb344d6f337e
(no functionnal change)
2021-10-10 21:08:48 +02:00
Markus Heiser 2b69710aef [mod] improve video results of the qwant engine
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-10 21:08:48 +02:00
Markus Heiser 3205785059 [fix] Qwant engines - implement API v3 and add 'quant videos'
The implementation uses the Qwant API (https://api.qwant.com/v3). The API is
undocumented but can be reverse engineered by reading the network log of
https://www.qwant.com/ queries.

This implementation is used by different qwant engines in the settings.yml::

  - name: qwant
    categories: general
    ...
  - name: qwant news
    categories: news
    ...
  - name: qwant images
    categories: images
    ...
  - name: qwant videos
    categories: videos
    ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-10 21:08:48 +02:00
Allen 79dc10e382
[fix] Update about section of Invidious and Rumble + Change filtron error wording (#2959)
* [fix] Update about section of Invidious

Another website and new documentation

* [fix] Correct engine name in for Rumble (#11)

* [fix] Wording for Filtron error message (#12)
2021-10-10 16:44:21 +02:00
Simon Schuster 23d6c9c798 [fix] use engine-type when looking up supported_languages from JSON files
searx/data/engines_languages.json stores language information for
several searchengines in a json endoded dict that maps engine-"types" to
their supported languages; for instance there is an entry "google",
mapping to the supported languages of the google engine.

However, the lookup code did not use the engine 'type' (as in: the
filename searx/engines/<enginetype>.py), but instead the manually
configured engine name from settings.yml when querying. This is
problematic as soon as users start to specify additional engine
instances with custom names in the config file, as for instance
suggested as a workaround for multilingual search in the manual[0]:

> engines:
>   - name : google english
>     engine : google
>     language : english

Here, the engine name "google english" will be used for the lookup in
the json file, which does not exist. The empty supported_languages then
lead to a type error later in the processing callchain.

This patch changes the behaviour to use the engine's entry-"type"
("google" in the above example) for the lookup. This should fix bug #2928.

0: https://searx.github.io/searx/user/search_syntax.html#multilingual-search
2021-10-06 19:33:43 +02:00
Noémi Ványi 53c4031f96 [fix] minor style fixes after picking open street map changes 2021-10-02 15:05:03 +02:00
Markus Heiser eabdf1bae9 [fix] openstreetmap - fix some minor whitespace & indentation issues
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-02 14:59:01 +02:00
Alexandre Flament 84dcd3af56 [enh] openstreetmap / map template: improve results
implements ideas described in #69

* update the engine
* use wikidata
* update map.html template
2021-10-02 14:57:30 +02:00
Paul Alcock bb34685dfa
Add IMDB support (#2980)
Closes #1145
2021-10-02 13:41:38 +02:00
Noémi Ványi 1d5feed4c1 [fix] style of stackexchange engine 2021-10-02 13:25:50 +02:00
Markus Heiser a1f9919587 [fix] engine stackexchange - decode HTML entities in title & content
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-02 11:49:12 +02:00
Markus Heiser 5e84d670a2 [mod] replace old stackoverflow engine by Stack Exchange API v2.3
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-02 11:43:51 +02:00
Markus Heiser 2cf9a61246 [mod] engines - add Stack Exchange API v2.3
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-02 11:41:36 +02:00
Noémi Ványi 3f05513601 [fix] DDG engine format and remove logger 2021-10-02 11:40:56 +02:00
Markus Heiser 9d9a89c6ef [mod] engine duckduckgo - update supported_languages_url
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-02 11:33:21 +02:00
Markus Heiser c218fb76b8 [mod] engine duckduckgo - use DuckDuckGo-Lite
Implement a scrapper for DuckDuckGo-Lite [1].  The existing DuckDuckGo [2]
engine does not support paging.  DuckDuckgo-Lite is much faster, less verbose
and does have a paging option (reversed engineered from the input form of [1]).

[1] https://lite.duckduckgo.com/lite
[2] https://duckduckgo.com/

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-02 11:33:12 +02:00
Markus Heiser 987b0d6f5b [fix] remove minimum length of content for XPath engine
Instead of raising an exception and therefore hiding all results of the engine.

It make sense to remove that requirement in order to allow the implementation of
search engines that do not always have a description.  In fact some search
engines that in 99% of the case have a description like Brave Search or Mojeek
crash completely if they for some reason included a result with no description.

To test this patch try Mojeek:

    !mjk xyz

before and after the patch.

Suggested-by: 0xhtml in https://github.com/searx/searx/discussions/2933
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-13 21:24:26 +02:00
dependabot[bot] e271d6d1e1 Bump pylint from 2.9.6 to 2.10.2
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.9.6 to 2.10.2.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.9.6...v2.10.2)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-08-22 20:53:12 +02:00
Noémi Ványi 23b3b56a06 add filter=0 to Google engine for more results
Closes #2944
2021-08-21 16:29:20 +02:00
Émilien Devos ee443d9739 Fix google images
Proposed fix in https://github.com/searx/searx/pull/2115#issuecomment-876716010

Closes #2914
2021-08-02 20:14:54 +02:00
Samuel Dudík 13a608b0b8
Fix Seznam engine (#2905)
## What does this PR do?

Fixes the Seznam engine by updating XPath strings.

## Why is this change important?

Without this PR Seznam returns no results.

Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
2021-08-02 20:09:31 +02:00
Marc Abonce Seguin a5839a66d6
Update onion engines to v3 (#2904)
downgrade httpx:
PR https://github.com/encode/httpx/pull/1522
made some breaking changes in AsyncHTTPTransport that affect
our code in https://github.com/searx/searx/blob/master/searx/network/client.py

remove not_evil which has been down for a while now:
https://old.reddit.com/r/onions/search/?q=not+evil&restrict_sr=on&t=year
2021-08-02 20:03:55 +02:00