Final commit with the latest changes in the project files
This commit is contained in:
Alexey Nizhegolenko
parent
9ef949a704
commit
7dba8e3967
4 changed files with 44 additions and 22 deletions
62
README.md
62
README.md
|
|
@ -1,15 +1,26 @@
|
|||
# GeoStat
|
||||
### Version 1.0
|
||||
### Version 2.0
|
||||
|
||||
|
||||
|
||||
GeoStat is a Python script for parsing Nginx logs files and getting GEO data from incoming IP's in it. This script converts parsed data into JSON format and sends it to the InfluxDB database so you can use it to build some nice Grafana dashboards for example. It runs as service by SystemD and parses log in tailf command style. Also, it can be run as a Docker container for the easy start.
|
||||
GeoStat it's a Python-based script for parsing Nginx and Apache log files and getting GEO data from incoming IPs from it. This script converts parsed data into JSON format and sends it to the InfluxDB database, so you can use it for building nice Grafana dashboards for example. The application runs as SystemD service and parses log files in "tailf" style. Also, you can run it as a Docker container if you wish.
|
||||
|
||||
# New in version 2.0
|
||||
- The application was rewritten with python3
|
||||
- Was added few additional tags to JSON output, the country name, and the city name.
|
||||
- Was fixed the few bugs with the geohash lib and with the install.sh script.
|
||||
- Also was added the simple logging feature, now you can catch this application log in the Syslog file.
|
||||
- The Dockerfile was recreated with the python3 support also.
|
||||
- Was done all needed tests, and all things looks OK :)
|
||||
|
||||
# Main Features:
|
||||
|
||||
- Parsing incoming IPS from web server log and convert them into GEO metrics for the InfluxDB.
|
||||
- Used standard python libs for maximum compatibility.
|
||||
- Having an external **settings.ini** for comfortable changing parameters.
|
||||
- Have a Docker file for quick building Docker image.
|
||||
- Parsing incoming IPs from web server log and converts them into GEO metrics for the InfluxDB.
|
||||
- Using standard python libs for maximum easy use.
|
||||
- Having an external **settings.ini** file for comfortable changing parameters.
|
||||
- Having a Dockerfile inside for quick building Docker image.
|
||||
- Contains an install.sh script that will do the installation process easy.
|
||||
- Runs as a SystemD service
|
||||
|
||||
JSON format that script sends to InfluxDB looks like:
|
||||
```
|
||||
|
|
@ -23,33 +34,42 @@ JSON format that script sends to InfluxDB looks like:
|
|||
'host': 'cube'
|
||||
'geohash': 'u8mb76rpv69r',
|
||||
'country_code': 'UA'
|
||||
'country_name': 'Ukraine'
|
||||
'city_name': 'Odessa'
|
||||
}
|
||||
}
|
||||
]
|
||||
```
|
||||
As you can see there are three tags fields, so you can build dashboards using geohash (with a point on the map) or country code, or build dashboards with variables based on the host name tag. A count for any metric equals 1. This script doesn't parse log file from the beginning but parses it line by line after running. So you can build dashboards using **count** of geohashes or country codes after some time will pass.
|
||||
As you can see there are five fields in the JSON output, so you can build dashboards using geo-hash (with a point on the map) or country code, or with the country name and city name. Build dashboards with variables based on the hostname tag or combine them all. A count for any metric equals 1, so it'll be easy summarising. This script doesn't parse the log file from the beginning but parses it line by line after starting. So you can build dashboards using **count** of data after some time will pass.
|
||||
|
||||
You can find the example Grafana dashboard in **geomap.json** file or from grafana.com: https://grafana.com/dashboards/8342
|
||||
You can find the example of the Grafana dashboard in **geomap.json** file or take it from the grafana.com: https://grafana.com/dashboards/8342
|
||||
|
||||
### Tech
|
||||
|
||||
GeoStat uses a number of open source libs to work properly:
|
||||
|
||||
* [Geohash](https://github.com/vinsci/geohash) - Python module that provides functions for decoding and encoding Geohashes.
|
||||
* [Geohash](https://github.com/vinsci/geohash) - Python module that provides functions for decoding and encoding Geohashes. Now it was added as local lib, and no longer need to be installed with pip.
|
||||
* [InfluxDB-Python](https://github.com/influxdata/influxdb-python) - Python client for InfluxDB.
|
||||
|
||||
## Important
|
||||
The GeoLite2-City database no longer available for simple downloading, you need to register on the maxmind.com website After you get an account on the maxmind.com you can find the needed file by the link (https://www.maxmind.com/en/accounts/YOURACCOUNTID/geoip/downloads)
|
||||
Please don't forget to unzip and put the GeoLite2-City.mmdb file in the same directory with the geoparse.py script, or you can put it anywhere and then fix the path in the settings.ini.
|
||||
|
||||
# Installation
|
||||
|
||||
You can install it in a few ways:
|
||||
|
||||
Using install.sh script:
|
||||
1) Clone the repository.
|
||||
2) CD into dir and run **install.sh**, it will ask you to set a properly settings.ini parameters, like Nginx **access.log** path, and InfluxDB settings.
|
||||
3) After the script will finish you only need to start SystemD service with **systemctl start geostat.service**.
|
||||
2) CD into the directory and then run **install.sh**, it will asks you to set properly settings.ini parameters, like Nginx/Apache **access.log** path, and InfluxDB settings.
|
||||
3) After the script will finish the application installationion you need copy the GeoLite2-City.mmdb file into the application local directory and start the SystemD service with **systemctl start geostat.service**.
|
||||
|
||||
Manually:
|
||||
1) Clone the repository, create an environment and install requirements
|
||||
```sh
|
||||
$ cd geostat
|
||||
$ virtualenv venv && source venv/bin/activate
|
||||
$ pip install -r requirements.txt
|
||||
$ python3 -m venv venv && source venv/bin/activate
|
||||
$ pip3 install -r requirements.txt
|
||||
```
|
||||
2) Modify **settings.ini** & **geostat.service** files and copy service to systemd.
|
||||
```sh
|
||||
|
|
@ -59,11 +79,9 @@ $ cp geostat.service.template geostat.service
|
|||
$ vi geostat.service
|
||||
$ cp geostat.service /lib/systemd/system/
|
||||
```
|
||||
3) Download latest GeoLite2-City.mmdb from MaxMind
|
||||
3) Register and download latest GeoLite2-City.mmdb file from maxmind.com
|
||||
```sh
|
||||
$ wget https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
|
||||
$ tar -xvzf GeoLite2-City.tar.gz
|
||||
$ cp ./GeoLite2-City_some-date/GeoLite2-City.mmdb ./
|
||||
$ cp ./any_path/GeoLite2-City.mmdb ./
|
||||
```
|
||||
4) Then enable and start service
|
||||
```sh
|
||||
|
|
@ -71,17 +89,21 @@ $ systemctl enable geostat.service
|
|||
$ systemctl start geostat.service
|
||||
```
|
||||
Using Docker image:
|
||||
1) Build the docker image from the Dockerfile inside geostat repository directory run:
|
||||
1) Build the docker image using the Dockerfile inside geostat repository directory:
|
||||
```
|
||||
$ docker build -t some-name/geostat .
|
||||
```
|
||||
2) After Docker image will be created you can run it using properly edited **settings.ini** file and you also,
|
||||
2) Register and download latest GeoLite2-City.mmdb file from maxmind.com
|
||||
```sh
|
||||
$ cp ./any_path/GeoLite2-City.mmdb ./
|
||||
```
|
||||
3) After Docker image will be created you can run it using properly edited **settings.ini** file and you also,
|
||||
need to forward the Nginx/Apache logfile inside the container:
|
||||
```
|
||||
docker run -d --name geostat -v /opt/geostat/settings.ini:/settings.ini -v /var/log/nginx_access.log:/var/log/nginx_access.log some-name/geostat
|
||||
```
|
||||
|
||||
After the first metrics will go to the InfluxDB you can create nice Grafana dashboards.
|
||||
After the first metrics will reach the InfluxDB you can create nice dashboards in Grafana.
|
||||
|
||||
Have fun !
|
||||
|
||||
|
|
|
|||
|
|
@ -88,7 +88,7 @@ def logparse(LOGPATH, INFLUXHOST, INFLUXPORT, INFLUXDBDB, INFLUXUSER, INFLUXUSER
|
|||
try:
|
||||
CLIENT.write_points(METRICS)
|
||||
except Exception:
|
||||
logging.exception("Cannot establish connection to InfluxDB: ") # NOQA
|
||||
logging.exception("Cannot establish connection with InfluxDB server: ") # NOQA
|
||||
|
||||
|
||||
def main():
|
||||
|
|
|
|||
BIN
geostat.png
BIN
geostat.png
Binary file not shown.
|
Before Width: | Height: | Size: 113 KiB After Width: | Height: | Size: 188 KiB |
|
|
@ -36,7 +36,7 @@ echo "Last step, you need to register and download the lates GeoLite2 City mmdb
|
|||
echo "After you get an account on the maxmind.com you can find the needed file by the link below"
|
||||
echo "https://www.maxmind.com/en/accounts/YOURACCOUNTID/geoip/downloads"
|
||||
echo "Please don't forget to unzip and put the GeoLite2-City.mmdb file in the same directory with the geoparse.py"
|
||||
echo "script, or you can put it enywhere and then change the way in the settings.ini"
|
||||
echo "script, or you can put it enywhere and then fix the path in the settings.ini"
|
||||
|
||||
echo ""
|
||||
echo "Good, all was done and you can start getting GEO data from your Nginx/Apache log file now"
|
||||
|
|
|
|||
Loading…
Reference in a new issue