aka: ensure that 'make test' works as expected
The cache contains a copy './local' which is - under some circumstance -
corrupted. It is not possible to clear the cache [1] (see the top of the page).
Ensure that 'make test' works as expected [2] even if
- the python interpreter is missing
- the virtualenv exists but pyyaml is missing
To hardening when the workflow cache fails, this patch adds the new target
'travis.test' into the workflow. This target probes to import a python module
'yaml'. If this fails the virtualenv will be completely new build.
[1] https://github.com/actions/cache/issues/2#issuecomment-673493515
[2] https://github.com/searx/searx/pull/2517#discussion_r567240235
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Target pip-exe is a prerequisite of the targets:
- pyinstall
- pyuninstall
and was accidentally deleted in commit 9b48ae47.
HINT:
do not confuse pyinstall with penvinstall
pyinstall & pyuninstall
Installing into user's HOME using pip from OS,
therefore the message is needed.
pyenvinstall & pyenvuninstall
Installing into virtualenv (./local) using pip which is provided by
prerequisite 'pyenv' in the virtualenv.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1. This patch fixes error:
rm -rf gh-pages/
make V=1 gh-pages
make[1]: Leaving directory '/800GBPCIex4/share/searx'
[ -d "gh-pages/.git" ] || git clone gh-pages
fatal: repository 'gh-pages' does not exist
2. The gh-page build has been moved to ./build/gh-pages this also affects
'travis-gh-pages'
3. The gh-pages commit messages now includes a ref to the repository and commit
4. Since a gh-pages history has only the drawback that the reposetory grows
fast, this patch also flattens the history:
cd build/gh-pages/; git log --oneline
bash: cd: build/gh-pages/: Datei oder Verzeichnis nicht gefunden
026126be (HEAD -> gh-pages, origin/gh-pages) make gh-pages: from https://github.com/return42/searx.git@71d66979c2935312e0aed7fc7c3cf6199fbe88a2
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Avoid SearxEngineXPathException errors when parsing non valid results::
.//div[@class="yuRUbf"]//a/@href index 0 not found
Traceback (most recent call last):
File "./searx/engines/google.py", line 274, in response
url = eval_xpath_getindex(result, href_xpath, 0)
File "./searx/searx/utils.py", line 608, in eval_xpath_getindex
raise SearxEngineXPathException(xpath_spec, 'index ' + str(index) + ' not found')
searx.exceptions.SearxEngineXPathException: .//div[@class="yuRUbf"]//a/@href index 0 not found
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
BTW: fix indentation by 2 spaces
The additional tests has been commented out in the google engines to not release
any CAPTCHA issues.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
BTW: make the engines ready for search.checker:
- replace eval_xpath by eval_xpath_getindex and eval_xpath_list
- google_images: remove outer try/except block
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To check all engines:
make search.checker
To check a engine 'google news' replace space by underline:
make search.checker.google_news
To see HTTP requests and more use SEARX_DEBUG:
make SEARX_DEBUG=1 search.checker.google_news
To filter out HTTP redirects:
make SEARX_DEBUG=1 search.checker.google_news | grep -A1 "HTTP/1.1\" 3[0-9][0-9]"
...
Engine google news Checking
https://news.google.com:443 "GET /search?q=life&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
https://news.google.com:443 "GET /search?q=life&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
--
https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
--
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The 'video.html' template from the 'oscar' design supports replacement
for *author* and *length*. Google-videos does not have an author, alternatively
the publisher info from is used for the *author*.
Hint: these replacements are not supported by the 'simple' design.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This revise is based on the methods developed in the revise of the google engine
(see commit 410c2f9).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
$ make test.sh
In utils/lxc.sh line 42:
ubu2010_boilerplate="$ubu1904_boilerplate"
^-----------------^ SC2034: ubu2010_boilerplate appears unused. Verify use (or export if used externally).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This revise is based on the methods developed in the revise of the google engine
(see commit 410c2f9).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The fabfile.py has not been updated since 5 years. I also asked [1] if someone
still use Fabric wtihout any response. Lets drop outdated Fabric file.
[1] https://github.com/searx/searx/discussions/2400
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
pip install -e .
...
Obtaining file:///usr/local/searx/searx-src
ERROR: Command errored out with exit status 1:
command: /usr/local/searx/searx-pyenv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/usr/local/searx/searx-src/setup.py'"'"'; __file__='"'"'/usr/local/searx/searx-src/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'rn'"'"', '"'"'n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-vzer91m2
cwd: /usr/local/searx/searx-src/
Complete output (9 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/searx/searx-src/setup.py", line 10, in <module>
from searx.version import VERSION_STRING
File "/usr/local/searx/searx-src/searx/__init__.py", line 19, in <module>
import searx.settings_loader
File "/usr/local/searx/searx-src/searx/settings_loader.py", line 8, in <module>
import yaml
ModuleNotFoundError: No module named 'yaml'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Alexandre Flament
Markus Heiser
Adam Tauber
Tobi823
mrwormo