diff --git a/2019/Searx_Setup_With_Docker_and_filtron/rules.json_edited b/2019/Searx_Setup_With_Docker_and_filtron/rules.json_edited new file mode 100644 index 0000000..59e0dfb --- /dev/null +++ b/2019/Searx_Setup_With_Docker_and_filtron/rules.json_edited @@ -0,0 +1,66 @@ +[ + { + "name": "search request", + "filters": ["Param:q", "Path=^(/|/search)$"], + "interval": 60, + "limit": 8, + "subrules": [ + { + "name": "roboagent limit", + "interval": 60, + "limit": 8, + "filters": +["Header:User-Agent=(curl|cURL|Wget|python-requests|Scrapy|FeedFetcher|Go-http-client)"], + "actions": [ + {"name": "block", + "params": {"message": "Rate limit exceeded"}} + ] + }, + { + "name": "botlimit", + "limit": 0, + "stop": true, + "filters": +["Header:User-Agent=(Googlebot|bingbot|Baiduspider|yacybot|YandexMobileBot|YandexBot|Yahoo! +Slurp|MJ12bot|AhrefsBot|archive.org_bot|msnbot|MJ12bot|SeznamBot|linkdexbot|Netvibes|SMTBot|zgrab|James BOT)"], + "actions": [ + {"name": "block", + "params": {"message": "Rate limit exceeded"}} + ] + }, + { + "name": "IP limit", + "interval": 60, + "limit": 32, + "stop": true, + "aggregations": ["Header:X-Forwarded-For"], + "actions": [ + {"name": "block", + "params": {"message": "Rate limit exceeded"}} + ] + }, + { + "name": "rss/json limit", + "interval": 60, + "limit": 30, + "stop": true, + "filters": ["Param:format=(csv|json|rss)"], + "actions": [ + {"name": "block", + "params": {"message": "Rate limit exceeded"}} + ] + }, + { + "name": "useragent limit", + "interval": 60, + "limit": 8, + "aggregations": ["Header:User-Agent"], + "actions": [ + {"name": "block", + "params": {"message": "Rate limit exceeded"}} + ] + } + ] + } +] + diff --git a/2019/Searx_Setup_With_Docker_and_filtron/setup_log b/2019/Searx_Setup_With_Docker_and_filtron/setup_log new file mode 100644 index 0000000..61eea8c --- /dev/null +++ b/2019/Searx_Setup_With_Docker_and_filtron/setup_log @@ -0,0 +1,230 @@ +########################################## +NOTE: read all of this before starting. +Some steps were wrong... +########################################## + + + + +Setting up searx on kvm vps w/docker: + +This guide assumes you have Docker already installed (if not follow the official documentation) +This guide uses Devuan Ascii +This guide is built on a VPS with KVM / Docker support. Note that many VPS (w/openVZ at the moment) +do NOT support docker. +It is helpful to have some experience with Docker. Buy a book, if you have not read one already. +Other GNU/Linux Admin experience is assumed. +This guide is meant as an adjunct to the official Searx Documentation on installing. Read that, also. + + + +Let's begin. + + +First, clone the searx repo. + +cd into the directory. + +I had to make sure I pulled a specific tag release for searx + +otherwise the dockerfile build would fail. + + +git checkout tags/v0.14.0 + +then + + +sudo docker build -t whatever/searx . + +here, you are building the dockerfile in the same directory and giving it a name: whatever/searx (which obv can be +customized) + + + + +see searx is available with +docker images + +(at any time, you can type just docker, and it will list options) + + +docker run -d --name searx -p $PORT:8888 whatever/searx + +here $PORT will pick a random port for searx to be listening on + + +Test it works by viewing WANIP:PORT + +and it should work. + +problems: + +1) searx has bing and default search engines +2) no https +3) no filter, to block spammers (we need to use the filter asciimoo made, or make our own...) + + + +Most important is 3, followed by 1, and 2, in that order. + +Also need to give it port 80, or port 443. (probably need reverse proxy for nginx or something) +let's get a filter first. + +filtron is the filter. + +filtron sits between nginx and searx. +nginx -> filtron -> searx + + +https://asciimoo.github.io/searx/admin/filtron.html + + +good news is, filtron is managed by package manager in go. + +first install go. + +i'm using devuan ascii so, + +apt-get install golang + + +everyone online tells you to dl binary + + bad idea. + +slower, and unable to update. I am not dealing with un-updateable binaries. + + + +after you apt-get install need to set gopath + + + +put these two in /etc/profile: + +export PATH=$PATH:/usr/local/go/bin +export GOPATH=/root/go + +in debian, go has a path in /usr/share +which has pkg, src, test +that is the GOROOT +not the GOPATH + +so make something different for gopath +Typically it is a folder in users Home directory. + +I had an error +package math/bits: unrecognized import path "math/bits" (import path does not begin with hostname) + + +Go version + +and forums show that mine is too old. I COULD use the binary, but that's not what I'm going to do. + +EDIT: let's try backports first.... + +apt-get -t ascii-backports install golang + +that is 1.10 +not 1.7 + + +and that worked. +So you MUST use ascii-backports for this. + +OK. + + + + + +so install it. get the rules.json in this folder as an example. Note that the default, requires +you to set some variables + +run it with $GOPATH/bin/filtron -rules rules.json + +we will want to have it in the background, so something like above in rc.local (no service?) +with the & afterwards, perhaps. + + + + +So with filtron. +we want to organize like this + + +WAN +nginx ----> filtron -----> docker ------> searx +443 4004 $PORT 8888 + +We'll need to specify the docker port, and the filtron port, and the nginx port. +let's use 20000 as docker port. + +ignoring nginx, we have + +filtron --help shows us + + +FILTRON +============ +filtron -listen "127.0.0.1:4004" -target "127.0.0.1:20000" -rules rules.json + +NOTE: There is a trap with filtron. It expects a string, so -listen "127.0.0.1:4444" will work, but +simply typing -listen 4444 will NOT work. + +Verify filtron is listening with ss -ntlp, where you should see: + +LISTEN 0 128 127.0.0.1:4005 *:* +users:(("filtron",pid=27293,fd=3)) +LISTEN 0 128 127.0.0.1:4004 *:* +users:(("filtron",pid=27293,fd=5)) + +or similar. + + + +DOCKER +============ +docker run -d --name searx -p 20000:8888 whatever/searx + +searx is run by docker, and we don't need to worry about that. + +Then we need reverse ssl proxy for nginx. + +I can get that from the gitea page so I checked there first, and then here +https://nginx.org/en/docs/http/configuring_https_servers.html + +lets encrypt will be later. (I have that covered in my own lets encrypt docs) + + +so open a screen to test and run those, with & for filtron, docker will detatch with -d + +add +location / { + proxy_pass http://localhost:4004; + } + +or just the proxy pass part to the nginx config. (make a copy of default, and edit the copy, add symbolic link +to sites-enabled) + + +And if you want to troubleshoot, you can do it step by step with the above example of reverse - reverse - reverse proxy to searx.... + + +wget the docker ip at 20000 +wget the filtron ip 4004 +wget the nginx at 80 + +should all work. + + + + +That's it. + + + + + + diff --git a/README.md b/README.md index d0c7cdf..21e817d 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,11 @@ # IT_Articles -Articles I've written for customers on IT issues. +Articles I've written for customers on IT issues, or for general use online. #2018 Winmail Error Honeywell T6 Pro review + +#2019 +Setting up Searx with Filtron, docker, and an Nginx reverse proxy on Devuan Ascii.