sudo apt install telegraf<\/code><\/mark><\/p>\n\n\n\nadjust telegraf.conf to receive data from mqtt (and system metrics if desired) and send to victoria metrics (which can accept influxdb line protocol, so is a drop-in, low memory replacement for influxdb)<\/p>\n\n\n\n
sudo nano \/etc\/telegraf\/telegraf.conf<\/code><\/mark><\/p>\n\n\n\n# Telegraf Configuration\n#\n\n# Global tags can be specified here in key="value" format.\n[global_tags]\n # dc = "us-east-1" # will tag all metrics with dc=us-east-1\n # rack = "1a"\n ## Environment variables can be used as tags, and throughout the config file\n # user = "$USER"\n\n# Configuration for telegraf agent\n[agent]\n ## Default data collection interval for all inputs\n interval = "10s"\n ## Rounds collection interval to 'interval'\n ## ie, if interval="10s" then always collect on :00, :10, :20, etc.\n round_interval = true\n\n ## Telegraf will send metrics to outputs in batches of at most\n ## metric_batch_size metrics.\n ## This controls the size of writes that Telegraf sends to output plugins.\n metric_batch_size = 1000\n\n ## Maximum number of unwritten metrics per output. Increasing this value\n ## allows for longer periods of output downtime without dropping metrics at the\n ## cost of higher maximum memory usage.\n metric_buffer_limit = 10000\n\n ## Collection jitter is used to jitter the collection by a random amount.\n ## Each plugin will sleep for a random time within jitter before collecting.\n ## This can be used to avoid many plugins querying things like sysfs at the\n ## same time, which can have a measurable effect on the system.\n collection_jitter = "0s"\n\n ## Default flushing interval for all outputs. Maximum flush_interval will be\n ## flush_interval + flush_jitter\n flush_interval = "10s"\n ## Jitter the flush interval by a random amount. This is primarily to avoid\n ## large write spikes for users running a large number of telegraf instances.\n ## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s\n flush_jitter = "0s"\n\n ## Precision will NOT be used for service inputs. It is up to each individual\n ## service input to set the timestamp at the appropriate precision.\n precision = "0s"\n\n ## Log at debug level.\n debug = true\n ## Log only error level messages.\n # quiet = false\n\n ## Log target controls the destination for logs and can be one of "file",\n ## "stderr" or, on Windows, "eventlog". When set to "file", the output file\n ## is determined by the "logfile" setting.\n logtarget = "file"\n\n ## Name of the file to be logged to when using the "file" logtarget. If set to\n ## the empty string then logs are written to stderr.\n logfile = "\/var\/log\/telegraf\/telegraf.log"\n\n ## The logfile will be rotated when it becomes larger than the specified\n ## size. When set to 0 no size based rotation is performed.\n logfile_rotation_max_size = "1MB"\n\n ## Maximum number of rotated archives to keep, any older logs are deleted.\n ## If set to -1, no archives are removed.\n logfile_rotation_max_archives = 5\n\n ## Pick a timezone to use when logging or type 'local' for local time.\n ## Example: America\/Chicago\n log_with_timezone = "local"\n\n ## Override default hostname, if empty use os.Hostname()\n hostname = ""\n ## If set to true, do no set the "host" tag in the telegraf agent.\n omit_hostname = false\n\n###############################################################################\n# OUTPUT PLUGINS #\n###############################################################################\n[[outputs.influxdb]]\nurls = ["http:\/\/127.0.0.1:8428"]\n# # Configuration for sending metrics to InfluxDB\n# [[outputs.influxdb]]\n# ## The full HTTP or UDP URL for your InfluxDB instance.\n# ##\n# ## Multiple URLs can be specified for a single cluster, only ONE of the\n# ## urls will be written to each interval.\n# # urls = ["unix:\/\/\/var\/run\/influxdb.sock"]\n# # urls = ["udp:\/\/127.0.0.1:8089"]\n# urls = ["http:\/\/127.0.0.1:8086"]\n\n# ## The target database for metrics; will be created as needed.\n# ## For UDP url endpoint database needs to be configured on server side.\n database = "telegraf"\n##\n# ## If true, no CREATE DATABASE queries will be sent. Set to true when using\n# ## Telegraf with a user without permissions to create databases or when the\n# ## database already exists.\n skip_database_creation = false\n#\n# ## Name of existing retention policy to write to. Empty string writes to\n# ## the default retention policy. Only takes effect when using HTTP.\n# # retention_policy = ""\n#\n# ## HTTP Basic Auth\n username = "telegraf"\n password = "password"\n#\n###############################################################################\n# INPUT PLUGINS #\n###############################################################################\n[[inputs.cpu]]\npercpu = false\n#[[inputs.disk]]\n#ignore_fs = [ "tmpfs", "devtmpfs" ]\n#[[inputs.diskio]]\n#[[inputs.kernel]]\n[[inputs.mem]]\n#[[inputs.processes]]\n#[[inputs.swap]]\n[[inputs.system]]\n[[inputs.net]]\nfieldpass = [ "bytes*" ]\n[[inputs.netstat]]\n[[inputs.temp]]\n\n#This section reads correct system uptime to VictoriaMetrics so grafana can graph it correctly. \n#Normally telegraf [system] input refers to the build date\/time on startup, which is months out of date.\n[[inputs.exec]]\n commands = ["sh -c \\"awk '{print \\\\\\"system uptime=\\\\\\" int($1) \\\\\\"i\\\\\\"}' \/proc\/uptime\\""]\n timeout = "5s"\n data_format = "influx"\n name_override = "system_correct" # Different measurement name than normal system_uptime\n\n[[inputs.mqtt_consumer]]\nalias = "mqtt"\nstartup_error_behavior = "retry"\nservers = ["tcp:\/\/127.0.0.1:1883"]\ntopics = [\n "N\/0281f08bbe75\/system\/0\/#",\n ]\nclient_id = "telegraf"\nusername = "username"\npassword = "password"\ndata_format = "json"<\/code><\/pre>\n\n\n\nTelegraf is a memory hog and chokes with too many MQTT metrics, so it’s often easier to just capture a few metrics with Node Red and send directly to Victoria Metrics. Leaving telegraf uninstalled allows enough RAM to run a Grafana server. (an old version that does simple graphs, and uses MUCH less RAM). Another option is to install an old version of telegraf that also uses much less RAM.<\/p>\n\n\n\n
wget https:\/\/dl.influxdata.com\/telegraf\/releases\/telegraf_1.8.3-1_arm64.deb<\/code><\/mark><\/p>\n\n\n\nsudo dpkg -i telegraf_1.8.3-1_arm64.deb<\/code><\/mark><\/p>\n\n\n\n# Telegraf Configuration\n#\n\n# Global tags can be specified here in key="value" format.\n[global_tags]\n # dc = "us-east-1" # will tag all metrics with dc=us-east-1\n # rack = "1a"\n ## Environment variables can be used as tags, and throughout the config file\n # user = "$USER"\n\n# Configuration for telegraf agent\n[agent]\n ## Default data collection interval for all inputs\n interval = "10s"\n ## Rounds collection interval to 'interval'\n ## ie, if interval="10s" then always collect on :00, :10, :20, etc.\n round_interval = true\n\n ## Telegraf will send metrics to outputs in batches of at most\n ## metric_batch_size metrics.\n ## This controls the size of writes that Telegraf sends to output plugins.\n metric_batch_size = 1000\n\n ## Maximum number of unwritten metrics per output. Increasing this value\n ## allows for longer periods of output downtime without dropping metrics at the\n ## cost of higher maximum memory usage.\n metric_buffer_limit = 10000\n\n ## Collection jitter is used to jitter the collection by a random amount.\n ## Each plugin will sleep for a random time within jitter before collecting.\n ## This can be used to avoid many plugins querying things like sysfs at the\n ## same time, which can have a measurable effect on the system.\n collection_jitter = "0s"\n\n ## Default flushing interval for all outputs. Maximum flush_interval will be\n ## flush_interval + flush_jitter\n flush_interval = "10s"\n ## Jitter the flush interval by a random amount. This is primarily to avoid\n ## large write spikes for users running a large number of telegraf instances.\n ## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s\n flush_jitter = "0s"\n\n ## Precision will NOT be used for service inputs. It is up to each individual\n ## service input to set the timestamp at the appropriate precision.\n precision = "0s"\n\n ## Log at debug level.\n debug = true\n ## Log only error level messages.\n # quiet = false\n\n ## Log target controls the destination for logs and can be one of "file",\n ## "stderr" or, on Windows, "eventlog". When set to "file", the output file\n ## is determined by the "logfile" setting.\n # logtarget = "file"\n\n ## Name of the file to be logged to when using the "file" logtarget. If set to\n ## the empty string then logs are written to stderr.\n # logfile = "\/var\/log\/telegraf\/telegraf.log"\n\n ## The logfile will be rotated when it becomes larger than the specified\n ## size. When set to 0 no size based rotation is performed.\n # logfile_rotation_max_size = "1MB"\n\n ## Maximum number of rotated archives to keep, any older logs are deleted.\n ## If set to -1, no archives are removed.\n # logfile_rotation_max_archives = 5\n\n ## Pick a timezone to use when logging or type 'local' for local time.\n ## Example: America\/Chicago\n # log_with_timezone = "local"\n\n ## Override default hostname, if empty use os.Hostname()\n hostname = ""\n ## If set to true, do no set the "host" tag in the telegraf agent.\n omit_hostname = false\n\n###############################################################################\n# OUTPUT PLUGINS #\n###############################################################################\n[[outputs.influxdb]]\nurls = ["http:\/\/127.0.0.1:8428"]\n# # Configuration for sending metrics to InfluxDB\n# [[outputs.influxdb]]\n# ## The full HTTP or UDP URL for your InfluxDB instance.\n# ##\n# ## Multiple URLs can be specified for a single cluster, only ONE of the\n# ## urls will be written to each interval.\n# # urls = ["unix:\/\/\/var\/run\/influxdb.sock"]\n# # urls = ["udp:\/\/127.0.0.1:8089"]\n# urls = ["http:\/\/127.0.0.1:8086"]\n\n# ## The target database for metrics; will be created as needed.\n# ## For UDP url endpoint database needs to be configured on server side.\n database = "telegraf"\n##\n# ## If true, no CREATE DATABASE queries will be sent. Set to true when using\n# ## Telegraf with a user without permissions to create databases or when the\n# ## database already exists.\n skip_database_creation = false\n#\n# ## Name of existing retention policy to write to. Empty string writes to\n# ## the default retention policy. Only takes effect when using HTTP.\n# # retention_policy = ""\n#\n# ## HTTP Basic Auth\n username = "telegraf"\n password = "password"\n#\n###############################################################################\n# INPUT PLUGINS #\n###############################################################################\n[[inputs.cpu]]\npercpu = true\n[[inputs.disk]]\nignore_fs = [ "tmpfs", "devtmpfs" ]\n#[[inputs.diskio]]\n#[[inputs.kernel]]\n[[inputs.mem]]\n#[[inputs.processes]]\n#[[inputs.swap]]\n[[inputs.system]]\n[[inputs.net]]\nfieldpass = [ "bytes*" ]\n[[inputs.netstat]]\n[[inputs.temp]]\n\n#This section reads correct system uptime to VictoriaMetrics so grafana can graph it correctly. \n#Normally telegraf [system] input refers to the build date\/time on startup, which is months out of date.\n[[inputs.exec]]\n commands = ["sh -c \\"awk '{print \\\\\\"system uptime=\\\\\\" int($1) \\\\\\"i\\\\\\"}' \/proc\/uptime\\""]\n timeout = "5s"\n data_format = "influx"\n name_override = "system_correct" # Different measurement name than normal system_uptime\n\n[[inputs.mqtt_consumer]]\n#alias = "mqtt"\n#startup_error_behavior = "retry"\nservers = ["tcp:\/\/127.0.0.1:1883"]\ntopics = [\n "N\/0281f08bbe75\/system\/0\/#",\n ]\nclient_id = "telegraf"\nusername = "username"\npassword = "password"\ndata_format = "json"<\/code><\/pre>\n\n\n\nNote the [inputs.exec] portion, which pulls uptime from \/proc\/uptime, which is better than where telegraf normally gets it from (necessary because the system clock is updated a few minutes after boot using the modem).<\/p>\n\n\n\n
\n\n\n\nsudo apt-get install -y adduser libfontconfig1 musl<\/code><\/mark><\/p>\n\n\n\nwget https:\/\/dl.grafana.com\/enterprise\/release\/grafana-enterprise_6.7.0_arm64.deb<\/code><\/mark><\/p>\n\n\n\nsudo dpkg -i grafana-enterprise_6.7.0_arm64.deb<\/code><\/mark><\/p>\n\n\n\n
\n\n\n\nAnother light option is Perses, but has issues with CORS. <\/p>\n\n\n\n
wget https:\/\/github.com\/perses\/perses\/releases\/download\/v0.53.0-beta.2\/perses_0.53.0-beta.2_linux_arm64.tar.gz<\/code><\/mark><\/p>\n\n\n\nsudo tar -xvzf ~\/perses_0.53.0-beta.2_linux_arm64.tar.gz -C \/opt\/perses<\/code><\/mark><\/p>\n\n\n\n
\n\n\n\ninstall pivpn<\/p>\n\n\n\n
curl -L https:\/\/install.pivpn.io | bash<\/code><\/mark><\/p>\n\n\n\nNeed to create the tun0 device on each reboot with sudo nano ~\/maketun.sh<\/code><\/mark><\/p>\n\n\n\n#!\/bin\/bash\n\nmkdir -p \/dev\/net\nmknod \/dev\/net\/tun c 10 200\nchmod 600 \/dev\/net\/tun\n\nsystemctl restart openvpn@server<\/code><\/pre>\n\n\n\nsudo chmod 755 ~\/maketun.sh<\/code><\/mark><\/p>\n\n\n\nsudo crontab -e<\/code><\/mark><\/p>\n\n\n\nadd @reboot \/home\/user\/maketun.sh<\/code><\/mark><\/p>\n\n\n\n
\n\n\n\ncreate reverse ssh tunnel as detailed here<\/a>.<\/p>\n\n\n\n
\n\n\n\ninstall chrony and edit conf file<\/p>\n\n\n\n
sudo apt install chrony<\/code><\/mark><\/p>\n\n\n\nsudo nano \/etc\/chrony\/chrony.conf<\/code><\/mark><\/p>\n\n\n\n#add to the bottom\nallow 192.0.0.0\/8\nlocal stratum 8\nmanual<\/code><\/pre>\n\n\n\nThe ‘local stratum 8’ and ‘manual’ allows chrony to serve the system time, not just NTP sources. Useful for no network connectivity when you’re pulling system time from the cell phone tower using sudo mmcli -m 0 –time. <\/p>\n\n\n\n
Limit journal size<\/p>\n\n\n\n
sudo nano \/etc\/systemd\/journald.conf<\/code><\/mark><\/p>\n\n\n\n#add to bottom\nSystemMaxUse=50M\nRuntimeMaxUse=50M<\/code><\/pre>\n\n\n\nsudo timedatectl set-timezone Africa\/Dakar<\/code><\/mark><\/p>\n\n\n\n
\n\n\n\nSmall script to restart after 1 day based on uptime rather than a specific time. Can get into a boot loop otherwise.<\/p>\n\n\n\n
sudo nano \/usr\/local\/bin\/check_uptime_reboot.sh<\/p>\n\n\n\n
#!\/bin\/bash\n# Save as \/usr\/local\/bin\/check_uptime_reboot.sh\n\nUPTIME_SECONDS=$(cat \/proc\/uptime | awk '{print int($1)}')\n\nif [ "$UPTIME_SECONDS" -ge 86400 ]; then\n logger "System rebooting after $UPTIME_SECONDS seconds uptime"\n \/sbin\/reboot\nfi<\/code><\/pre>\n\n\n\nchmod +x \/usr\/local\/bin\/check_uptime_reboot.sh<\/p>\n\n\n\n
Put the script into crontab below…<\/p>\n\n\n\n
\n\n\n\nInstall cron so various things can be automated<\/p>\n\n\n\n
sudo apt install cron<\/code><\/mark><\/p>\n\n\n\nsudo crontab -e<\/code><\/mark><\/p>\n\n\n\nHas cron jobs for checking data balance using USSD commands (see script on post about using USSD commands)<\/p>\n\n\n\n
@reboot \/home\/user\/maketun.sh\n*\/5 * * * * \/usr\/local\/bin\/check_uptime_reboot.sh<\/code><\/pre>\n\n\n\n
\n\n\n\nIf you have a weak cell signal or the the stick is working hard, it can overheat causing reboots and general instability. One solution is to disable two of the four cores. Normal operations with this setup (apart from startup) idle at 0-10% cpu, so it’s not a big deal. Edit the rc.local file as follows:<\/p>\n\n\n\n
sudo nano \/etc\/rc.local<\/code><\/mark><\/p>\n\n\n\n#!\/bin\/bash\n\nmain() {\n cleanup_wwan\n disable_cpu_cores\n}\n\ndisable_cpu_cores() {\n echo 0 > \/sys\/devices\/system\/cpu\/cpu2\/online\n echo 0 > \/sys\/devices\/system\/cpu\/cpu3\/online\n}\n\ncleanup_wwan() {\n ip netns add null || true\n\n max_wait=20\n waited=0\n while [ $waited -lt $max_wait ]; do\n if ip link show wwan7 &>\/dev\/null; then\n break\n fi\n sleep 1\n waited=$((waited + 1))\n done\n\n for i in wwan1 wwan2 wwan3 wwan4 wwan5 wwan6 wwan7; do\n ip link set "$i" netns null\n done\n\n # this may be required on some boards to initialize the modem\n sleep 2\n if ! ip -4 addr show wwan0 | grep -q "inet "; then\n systemctl restart ModemManager\n fi\n}\n\nmain\necho heartbeat > \/sys\/devices\/platform\/leds\/leds\/red:power\/trigger\nexit 0<\/code><\/pre>\n\n\n\n
\n\n\n\nIf necessary, install fail2ban and configure. If only exposing OpenVPN port, not necessary.<\/p>\n\n\n\n
sudo apt install fail2ban<\/code><\/mark><\/p>\n\n\n\nsudo cp \/etc\/fail2ban\/jail.conf \/etc\/fail2ban\/jail.local<\/code><\/mark><\/p>\n\n\n\n