Browse Source

Searx.

master
Your Name 5 years ago
parent
commit
05668b8714
3 changed files with 300 additions and 1 deletions
  1. +66
    -0
      2019/Searx_Setup_With_Docker_and_filtron/rules.json_edited
  2. +230
    -0
      2019/Searx_Setup_With_Docker_and_filtron/setup_log
  3. +4
    -1
      README.md

+ 66
- 0
2019/Searx_Setup_With_Docker_and_filtron/rules.json_edited View File

@ -0,0 +1,66 @@
[
{
"name": "search request",
"filters": ["Param:q", "Path=^(/|/search)$"],
"interval": 60,
"limit": 8,
"subrules": [
{
"name": "roboagent limit",
"interval": 60,
"limit": 8,
"filters":
["Header:User-Agent=(curl|cURL|Wget|python-requests|Scrapy|FeedFetcher|Go-http-client)"],
"actions": [
{"name": "block",
"params": {"message": "Rate limit exceeded"}}
]
},
{
"name": "botlimit",
"limit": 0,
"stop": true,
"filters":
["Header:User-Agent=(Googlebot|bingbot|Baiduspider|yacybot|YandexMobileBot|YandexBot|Yahoo!
Slurp|MJ12bot|AhrefsBot|archive.org_bot|msnbot|MJ12bot|SeznamBot|linkdexbot|Netvibes|SMTBot|zgrab|James BOT)"],
"actions": [
{"name": "block",
"params": {"message": "Rate limit exceeded"}}
]
},
{
"name": "IP limit",
"interval": 60,
"limit": 32,
"stop": true,
"aggregations": ["Header:X-Forwarded-For"],
"actions": [
{"name": "block",
"params": {"message": "Rate limit exceeded"}}
]
},
{
"name": "rss/json limit",
"interval": 60,
"limit": 30,
"stop": true,
"filters": ["Param:format=(csv|json|rss)"],
"actions": [
{"name": "block",
"params": {"message": "Rate limit exceeded"}}
]
},
{
"name": "useragent limit",
"interval": 60,
"limit": 8,
"aggregations": ["Header:User-Agent"],
"actions": [
{"name": "block",
"params": {"message": "Rate limit exceeded"}}
]
}
]
}
]

+ 230
- 0
2019/Searx_Setup_With_Docker_and_filtron/setup_log View File

@ -0,0 +1,230 @@
##########################################
NOTE: read all of this before starting.
Some steps were wrong...
##########################################
Setting up searx on kvm vps w/docker:
This guide assumes you have Docker already installed (if not follow the official documentation)
This guide uses Devuan Ascii
This guide is built on a VPS with KVM / Docker support. Note that many VPS (w/openVZ at the moment)
do NOT support docker.
It is helpful to have some experience with Docker. Buy a book, if you have not read one already.
Other GNU/Linux Admin experience is assumed.
This guide is meant as an adjunct to the official Searx Documentation on installing. Read that, also.
Let's begin.
First, clone the searx repo.
cd into the directory.
I had to make sure I pulled a specific tag release for searx
otherwise the dockerfile build would fail.
git checkout tags/v0.14.0
then
sudo docker build -t whatever/searx .
here, you are building the dockerfile in the same directory and giving it a name: whatever/searx (which obv can be
customized)
see searx is available with
docker images
(at any time, you can type just docker, and it will list options)
docker run -d --name searx -p $PORT:8888 whatever/searx
here $PORT will pick a random port for searx to be listening on
Test it works by viewing WANIP:PORT
and it should work.
problems:
1) searx has bing and default search engines
2) no https
3) no filter, to block spammers (we need to use the filter asciimoo made, or make our own...)
Most important is 3, followed by 1, and 2, in that order.
Also need to give it port 80, or port 443. (probably need reverse proxy for nginx or something)
let's get a filter first.
filtron is the filter.
filtron sits between nginx and searx.
nginx -> filtron -> searx
https://asciimoo.github.io/searx/admin/filtron.html
good news is, filtron is managed by package manager in go.
first install go.
i'm using devuan ascii so,
apt-get install golang
everyone online tells you to dl binary
bad idea.
slower, and unable to update. I am not dealing with un-updateable binaries.
after you apt-get install need to set gopath
put these two in /etc/profile:
export PATH=$PATH:/usr/local/go/bin
export GOPATH=/root/go
in debian, go has a path in /usr/share
which has pkg, src, test
that is the GOROOT
not the GOPATH
so make something different for gopath
Typically it is a folder in users Home directory.
I had an error
package math/bits: unrecognized import path "math/bits" (import path does not begin with hostname)
Go version
and forums show that mine is too old. I COULD use the binary, but that's not what I'm going to do.
EDIT: let's try backports first....
apt-get -t ascii-backports install golang
that is 1.10
not 1.7
and that worked.
So you MUST use ascii-backports for this.
OK.
so install it. get the rules.json in this folder as an example. Note that the default, requires
you to set some variables
run it with $GOPATH/bin/filtron -rules rules.json
we will want to have it in the background, so something like above in rc.local (no service?)
with the & afterwards, perhaps.
So with filtron.
we want to organize like this
WAN
nginx ----> filtron -----> docker ------> searx
443 4004 $PORT 8888
We'll need to specify the docker port, and the filtron port, and the nginx port.
let's use 20000 as docker port.
ignoring nginx, we have
filtron --help shows us
FILTRON
============
filtron -listen "127.0.0.1:4004" -target "127.0.0.1:20000" -rules rules.json
NOTE: There is a trap with filtron. It expects a string, so -listen "127.0.0.1:4444" will work, but
simply typing -listen 4444 will NOT work.
Verify filtron is listening with ss -ntlp, where you should see:
LISTEN 0 128 127.0.0.1:4005 *:*
users:(("filtron",pid=27293,fd=3))
LISTEN 0 128 127.0.0.1:4004 *:*
users:(("filtron",pid=27293,fd=5))
or similar.
DOCKER
============
docker run -d --name searx -p 20000:8888 whatever/searx
searx is run by docker, and we don't need to worry about that.
Then we need reverse ssl proxy for nginx.
I can get that from the gitea page so I checked there first, and then here
https://nginx.org/en/docs/http/configuring_https_servers.html
lets encrypt will be later. (I have that covered in my own lets encrypt docs)
so open a screen to test and run those, with & for filtron, docker will detatch with -d
add
location / {
proxy_pass http://localhost:4004;
}
or just the proxy pass part to the nginx config. (make a copy of default, and edit the copy, add symbolic link
to sites-enabled)
And if you want to troubleshoot, you can do it step by step with the above example of reverse - reverse - reverse proxy to searx....
wget the docker ip at 20000
wget the filtron ip 4004
wget the nginx at 80
should all work.
That's it.

+ 4
- 1
README.md View File

@ -1,8 +1,11 @@
# IT_Articles # IT_Articles
Articles I've written for customers on IT issues.
Articles I've written for customers on IT issues, or for general use online.
#2018 #2018
Winmail Error Winmail Error
Honeywell T6 Pro review Honeywell T6 Pro review
#2019
Setting up Searx with Filtron, docker, and an Nginx reverse proxy on Devuan Ascii.

Loading…
Cancel
Save