Searx.

2019/Searx_Setup_With_Docker_and_filtron/rules.json_edited

@@ -0,0 +1,66 @@
+[
+    {
+        "name": "search request",
+        "filters": ["Param:q", "Path=^(/|/search)$"],
+        "interval": 60,
+        "limit": 8,
+        "subrules": [
+            {
+                "name": "roboagent limit",
+                "interval": 60,
+                "limit": 8,
+                "filters": 
+["Header:User-Agent=(curl|cURL|Wget|python-requests|Scrapy|FeedFetcher|Go-http-client)"],
+                "actions": [
+                    {"name": "block",
+                     "params": {"message": "Rate limit exceeded"}}
+                ]
+            },
+            {
+                "name": "botlimit",
+                "limit": 0,
+                "stop": true,
+                "filters": 
+["Header:User-Agent=(Googlebot|bingbot|Baiduspider|yacybot|YandexMobileBot|YandexBot|Yahoo! 
+Slurp|MJ12bot|AhrefsBot|archive.org_bot|msnbot|MJ12bot|SeznamBot|linkdexbot|Netvibes|SMTBot|zgrab|James BOT)"],
+                "actions": [
+                    {"name": "block",
+                     "params": {"message": "Rate limit exceeded"}}
+                ]
+            },
+            {
+                "name": "IP limit",
+                "interval": 60,
+                "limit": 32,
+                "stop": true,
+                "aggregations": ["Header:X-Forwarded-For"],
+                "actions": [
+                    {"name": "block",
+                     "params": {"message": "Rate limit exceeded"}}
+                ]
+            },
+            {
+                "name": "rss/json limit",
+                "interval": 60,
+                "limit": 30,
+                "stop": true,
+                "filters": ["Param:format=(csv|json|rss)"],
+                "actions": [
+                    {"name": "block",
+                     "params": {"message": "Rate limit exceeded"}}
+                ]
+            },
+            {
+                "name": "useragent limit",
+                "interval": 60,
+                "limit": 8,
+                "aggregations": ["Header:User-Agent"],
+                "actions": [
+                    {"name": "block",
+                     "params": {"message": "Rate limit exceeded"}}
+                ]
+            }
+        ]
+    }
+]
+

2019/Searx_Setup_With_Docker_and_filtron/setup_log

@@ -0,0 +1,230 @@
+##########################################
+NOTE: read all of this before starting. 
+Some steps were wrong...
+##########################################
+
+
+
+
+Setting up searx on kvm vps w/docker:
+
+This guide assumes you have Docker already installed (if not follow the official documentation)
+This guide uses Devuan Ascii
+This guide is built on a VPS with KVM / Docker support. Note that many VPS (w/openVZ at the moment)
+do NOT support docker.
+It is helpful to have some experience with Docker. Buy a book, if you have not read one already.
+Other GNU/Linux Admin experience is assumed.
+This guide is meant as an adjunct to the official Searx Documentation on installing. Read that, also.
+
+
+
+Let's begin.
+
+
+First, clone the searx repo.
+
+cd into the directory.
+
+I had to make sure I pulled a specific tag release for searx
+
+otherwise the dockerfile build would fail.
+
+
+git checkout tags/v0.14.0
+
+then
+
+
+sudo docker build -t whatever/searx .
+
+here, you are building the dockerfile in the same directory and giving it a name: whatever/searx (which obv can be
+customized)
+
+
+
+
+see searx is available with
+docker images
+
+(at any time, you can type just docker, and it will list options)
+
+
+docker run -d --name searx -p $PORT:8888 whatever/searx
+
+here $PORT will pick a random port for searx to be listening on
+
+
+Test it works by viewing WANIP:PORT
+
+and it should work.
+
+problems:
+
+1) searx has bing and default search engines
+2) no https
+3) no filter, to block spammers (we need to use the filter asciimoo made, or make our own...)
+
+
+
+Most important is 3, followed by 1, and 2, in that order.
+
+Also need to give it port 80, or port 443. (probably need reverse proxy for nginx or something)
+let's get a filter first.
+
+filtron is the filter.
+
+filtron sits between nginx and searx.
+nginx -> filtron -> searx
+
+
+https://asciimoo.github.io/searx/admin/filtron.html
+
+
+good news is, filtron is managed by package manager in go.
+
+first install go.
+
+i'm using devuan ascii so,
+
+apt-get install golang
+
+
+everyone online tells you to dl binary
+
+					bad idea.
+
+slower, and unable to update. I am not dealing with un-updateable binaries.
+
+
+
+after you apt-get install need to set gopath
+
+
+
+put these two in /etc/profile:
+
+export PATH=$PATH:/usr/local/go/bin
+export GOPATH=/root/go
+
+in debian, go has a path in /usr/share
+which has pkg, src, test
+that is the GOROOT
+not the GOPATH
+
+so make something different for gopath
+Typically it is a folder in users Home directory.
+
+I had an error
+package math/bits: unrecognized import path "math/bits" (import path does not begin with hostname)
+
+
+Go version
+
+and forums show that mine is too old. I COULD use the binary, but that's not what I'm going to do.
+
+EDIT: let's try backports first....
+
+apt-get -t ascii-backports install golang
+
+that is 1.10
+not 1.7
+
+
+and that worked.
+So you MUST use ascii-backports for this.
+
+OK.
+
+
+
+
+
+so install it. get the rules.json in this folder as an example. Note that the default, requires
+you to set some variables
+
+run it with $GOPATH/bin/filtron -rules rules.json
+
+we will want to have it in the background, so something like above in rc.local (no service?)
+with the & afterwards, perhaps.
+
+
+
+
+So with filtron. 
+we want to organize like this
+
+
+WAN				
+nginx ---->   filtron -----> docker ------> searx
+443		4004          $PORT          8888
+
+We'll need to specify the docker port, and the filtron port, and the nginx port.
+let's use 20000 as docker port.
+
+ignoring nginx, we have
+
+filtron --help shows us
+
+
+FILTRON
+============
+filtron -listen "127.0.0.1:4004" -target "127.0.0.1:20000" -rules rules.json
+
+NOTE: There is a trap with filtron. It expects a string, so -listen "127.0.0.1:4444" will work, but
+simply typing  -listen 4444 will NOT work. 
+
+Verify filtron is listening with ss -ntlp, where you should see:
+
+LISTEN     0      128                               127.0.0.1:4005                                                  *:*
+users:(("filtron",pid=27293,fd=3))
+LISTEN     0      128                               127.0.0.1:4004                                                  *:*
+users:(("filtron",pid=27293,fd=5))
+
+or similar.
+
+
+
+DOCKER
+============
+docker run -d --name searx -p 20000:8888 whatever/searx
+
+searx is run by docker, and we don't need to worry about that.
+
+Then we need reverse ssl proxy for nginx.
+
+I can get that from the gitea page so I checked there first, and then here
+https://nginx.org/en/docs/http/configuring_https_servers.html
+
+lets encrypt will be later. (I have that covered in my own lets encrypt docs)
+
+
+so open a screen to test and run those, with & for filtron, docker will detatch with -d
+
+add 
+location / {
+        proxy_pass http://localhost:4004;
+    }
+
+or just the proxy pass part to the nginx config. (make a copy of default, and edit the copy, add symbolic link
+to sites-enabled)
+
+
+And if you want to troubleshoot, you can do it step by step with the above example of reverse - reverse - reverse proxy to searx....
+
+
+wget the docker ip at 20000
+wget the filtron ip 4004
+wget the nginx at 80
+
+should all work.
+
+
+
+
+That's it. 
+
+
+
+
+
+

README.md

@@ -1,8 +1,11 @@
 # IT_Articles
 
-Articles I've written for customers on IT issues.
+Articles I've written for customers on IT issues, or for general use online.
 
 
 #2018
 Winmail Error
 Honeywell T6 Pro review
+
+#2019
+Setting up Searx with Filtron, docker, and an Nginx reverse proxy on Devuan Ascii.