Linux Deployment Playbook

WELCOME to my Linux Deployment Playbook

This little project started out years ago when I was trying to learn how to get better at IT. I wanted a place to store important information that I would be able to recall later. From this, the Linux Deployment Playbook was born.

There are NOT just copy and pasted instructions. I HAVE deployed everything in this guide at some point in my network whether it push it is in a live or test environment.

  • Linux Deployment Playbook – LDP.v1
    The first version started very humbly in a little Microsoft Word document. It wasn’t very much outside of literally Linux commands and helpful things to add that would assist me with quick and easy deployments.
  • Linux Deployment Playbook – LDP.v2
    At this point in my career I started taking technical writing very seriously. I had started to learn how to use Sphinx and Read the Docs in order to write simple documentation that could be very presentable. It took all the tedious work out of making everything look pretty in MS Word and allowed me to focus more on the content. With this, my knowledge base grew ever faster. At this point, i started adding more things involving networking as well as coding and scripting while still retaining a lot of its Linux-based foundation. I also started to introduce topics like running docker on Linux and deploying services like apache, WordPress, Nextcloud, and Plex.
  • Internal Deployment and Configuration Guides
    This is where things took a dive. You see, Originally I had developed my Linux Deployment Playbook (version 1) alongside another set of documentation for my network. One guide was focused heavily on Linux, the other on networking. As the internal documentation grew, and the projects I began to take on started increase, I started dividing my internal documentation up. One of them was an Engineering Aid. While this was primarily designed as a guide to allow an outside user to have full knowledge about my internal systems, this started to become my new knowledge dumping point. That’s right, LDPv2 started to become neglected and just like that I reverted back to writing all my documentation in rich text documents again.
  • Linux Deployment Playbook – LDP.v3
    Up until now, most of my knowledge base has been kept mostly to myself, except for the few friends that I would share this with. Not to mention reverting back to MS Word felt like a massive downgrade and I needed to revive the once great Linux Deployment Playbook again! At the time I was already hosting a website (this one) and I figured that this was the perfect platform to host my new playbook. The advantage of this was my ability to showcase my experience and knowledge in one area. After all, that was the original purpose of this website, was to serve a a personal and professional representation of myself. Here, the Linux Deployment Playbook has extended FAR past what its original intention was, Linux. This now hosts a large (and random) knowledge base of projects, but lets be real its all mostly Docker.
  • Linux Deployment Playbook – LDPv4 [TBD]
    This will be the next upcoming release of my LDP. Currently I am still working on version 3 (here) and I have not moved onto plans on what a 4th version would look like. I am still working on adding more things here and getting this cleaned up while adding more content. Keep your eyes peeled for updates.

If you took the time to read this I want to thank you. Seriously, I appreciate it. I have put a lot of hard work into this and I hope that by doing this I can save someone months (or even years) of just splashing around in the shallow section of the pool and afford them the opportunity to just headfirst into the deep end.

Enjoy and please while you are here check out the rest of the website!

This is a list of items that I am currently testing, learning, or have already worked on but I am just in the phase of adding them to the Playbook.

Stin / stout / sterr

tee
sed

AWK

Fail2ban (Docker)

Reference: https://www.youtube.com/watch?v=Ha8NIAOsNvo
Reference: https://hub.docker.com/r/crazymax/fail2ban

VIM tutorial

REGEX

To-Do List with Vikunja (Docker)

Dashboard with Homer (Docker)

Using Low Code to Develop Internal Tools with Appsmith (Docker)

https://github.com/appsmithorg/appsmith

https://docs.appsmith.com/setup/docker

https://hub.docker.com/u/appsmith

Scanning Webapps with Hawkscan (Docker)

https://hub.docker.com/r/stackhawk/hawkscan

Reverse Proxy with Traefik (Docker)

System Monitoring with Prometheus (Docker)


Download Powered By Raspberry Pi - Raspberry Pi 3 B+ Logo PNG Image with No  Background - PNGkey.com

So you got a Raspberry Pi and are looking for some awesome projects to use with it. Well the Raspberry Pi is a small “credit card” shaped server that is praised for its small form factor and power (for its size). This tool is used to teach programming, robotics, network engineering, and security as well as a list of other things.

So what are some cool projects that you can do RIGHT NOW with your Raspberry Pi?

Install Docker on Raspberry Pi

Install script:

curl -fsSL https://get.docker.com -o get-docker.sh

Execute script:

sudo sh get-docker.sh

Install a Network Adblocker with Pi-hole (Docker)

Pi-hole – Network-wide protection

The Pi-hole is a POWERFUL system that can be used as a DNS server, DHCP server, and adblocker all in one.
Use Pi-hole to block advertisements on a network level.
Get useful network stats in an easy to understand graphical layout using the Pi-hole web interface.
Deploy instantly and use now!

Install Pi-hole with Docker (Recommended)

Pull the image:

docker pull pihole/pihole

Docker run:

docker run -d \
    --name pihole \
    -p 53:53/tcp -p 53:53/udp \
    -p 8080:80 \
    -v "${PIHOLE_BASE}/etc-pihole:/etc/pihole" \
    -v "${PIHOLE_BASE}/etc-dnsmasq.d:/etc/dnsmasq.d" \
    --dns=127.0.0.1 --dns=1.1.1.1 \
    --restart=unless-stopped \
    --hostname pi.hole \
    -e VIRTUAL_HOST="pi.hole" \
    -e PROXY_LOCATION="pi.hole" \
    -e ServerIP="127.0.0.1" \
    pihole/pihole:latest

Access the web interface for Pi-hole, simply type in the IP address and the port assigned or <ipaddr>/admin.

Install Pi-hole Bare Metal

Docker run command:

curl -sSL https://install.pi-hole.net | bash

To access the web interface for Pi-hole, simply type <ipaddr>/admin.

Reference: https://hub.docker.com/r/pihole/pihole
Reference: https://github.com/pi-hole/pi-hole

Install a Private VPN Server with PiVPN

For VPN access, it is recommended to install WireGuard for Docker on the Pi server.
PiVPN uses OpenVPN and/or WireGuard with custom commands for simple deployment and management.

Install:

curl -L https://install.pivpn.io | bash

Reference: https://github.com/pivpn/pivpn

Install a Honeypot with HoneyPi

HoneyPi: The smart beehive scale - Apps on Google Play

Download the zip file

wget https://github.com/mattymcfatty/HoneyPi/archive/master.zip

Unzip

unzip master.zip

Navigate into the directory

cd HoneyPi-master

Make the file executable

chmod +x *.sh

Execute the script

sudo ./honeyPiInstaller.sh

From here you just need to follow the prompts in the terminal in order to setup the rest of the honeypot

Reference: https://trustfoundry.net/honeypi-easy-honeypot-raspberry-pi/

Install Media Server with Plex (Docker)

Install Cloud Storage Server with Nextcloud (Docker)

QEMU/KVM

Savannah Wallpapers - Top Free Savannah Backgrounds - WallpaperAccess
Logo - QEMU

In virtualization a type-1 hypervisor is known as a “bare metal” installation in that the virtualization software runs directly on the hardware of the system unlike a type-2 hypervisor which is installed inside of an operating system. For this reason, type-1 hypervisors are more efficient, perform better and are often more secure compared to their type-2 counterparts. QEMU is installed and configured like a type-2 hypervisor (in that it is installed inside of an operating system). KVM is installed and operates like a type-1 hypervisor because it operates within the kernel giving direct access to system hardware.

The pitfalls of using a Type Two Hypervisor for running older Apps with WVD  – Ryan Mangan's IT Blog

I run KVM on Arch Linux as my Hypervisor. The reason I do this is because the Hypervisor OS should have the minimal software required to run QEMU/KVM/Libvirt. Arch Linux only comes with the packages that you choose to install making it bare minimum.

Install QEMU/KVM/Libvirt

Install KVM packages:

sudo pacman -Syy
sudo pacman -S archlinux-keyring
sudo pacman -S qemu virt-manager virt-viewer dnsmasq vde2 bridge-utils openbsd-netcat

Install libguestfs:

sudo pacman -S ebtables iptables

Start KVM libvirt service:

sudo systemctl enable libvirtd.service
sudo systemctl start libvirtd.service

Enable Nested Virtualization

Nested virtualizations allows you to run virtual machines inside of virtual machines. This is not supported by all hardware and is typically reserved for server motherboards.

Check to see if your motherboard supports nested virtualization:

sudo modprobe -r kvm_intel
sudo modprobe kvm_intel nested=1
echo "options kvm-intel nested=1" | sudo tee /etc/modprobe.d/kvm-intel.conf

Rename KVM Server

WARNING
Make sure that the Virtual Machine is shut down properly before proceeding

Rename the VM:

sudo virsh domrename <serverA> <serverB>

Resizing QEMU Images

WARNING
Make sure that the Virtual Machine is shut down properly before proceeding

Increase the size of a QCOW2 image:

sudo qemu-img resize image.qcow2 +SIZE

Shrinking a QCOW2 image is not as easy as increasing the size. It can be complicated and can cause damage to files residing on the image; For this matter, it is essential that you backup the image before attempting to resize:

sudo cp image.qcow2 backupimage.qcow2

Decrease the size of a QCOW2 image:

qemu-img create -f qcow2 -o preallocation=metadata newimage.qcow2 NEW_SIZE
virt-resize oldimage.qcow2 newimage.qcow2

Libvirt (Command Line Interface)

Daniel P. Berrangé » Blog Archive » New libvirt website design

Libvirt is a command line utility for managing virtual machine.

Here is the command format:

virsh [OPTION]... <command> <domain> [ARG]...

To list all host resources on the hypervisor:

virsh nodeinfo

Get a list of VM on the host:

#List active domains
virsh list 

#List all domains
virsh list --all

Rename a VM:

virsh domrename <current_name> <new_name>

Start a VM

#Start
virsh start <name>

#Enable autostart
virsh autostart <name>

#Disable autostart
virsh autostart  --disable <name>

Stop (shutdown) a VM

#Graceful shutdown
virsh shutdown <name>

#Force shutdown
virsh destroy <name>

Reboot a VM

virsh reboot <name>

Suspend a VM:

virsh suspend <name>

Resume a VM:

virsh resume <name>

Save a VM:

sudo virsh save <name> <name>.saved

Restore a VM:

sudo virsh restore <name>.save

Create a VM

sudo virt-install \
  --name <name> \
  --description "Enter Description" \
  --ram=<bits of RAM> \
  --vcpus=<number of CPUs> \
  --os-type=Linux \
  --os-variant=<distro> \
  --disk path=/path/to/file.qcow2,bus=virtio,size=<disk size> \
  --graphics none \
  --location /path/to/image.iso \
  --network bridge:<interface> \
  --console pty,target_type=serial -x 'console=ttyS0,115200n8 serial'

To edit the XML of a VM:

virsh edit <name>

Virtual Machine Manager VMM (Desktop User Interface)

virt-manager · GitHub

VMM is a desktop utility for managing virtual machines

Cockpit (Web User Interface)

cockpit-project · GitHub

Cockpit is a web user interface for managing a remote server.

Install Cockpit web interface (Ubuntu):

sudo apt update && sudo apt install cockpit

Install Cockpit web interface (Arch Linux):

sudo pacman -S cockpit && sudo systemctl enable --now cockpit.socket

Cockpit web interface is accessed over port 9090.

To access virtual machines in the Cockpit web UI, you need to install the “Cockpit-Machines” package

Cockpit-Machines (Ubuntu):

sudo apt update && sudo apt install cockpit-machines

Cockpit Machines (Arch)

sudo pacman -S cockpit-machines

Shared File System

You can mount directories from the host OS into a VM as fileshares with VIRTIOFS.

First you need to create a shared directory with a “source path” nd a “mount tag”.

Manually mount with Virtiofs:

sudo mount -t virtiofs <share> /mount/location

You can mount automatically by editing the fstab file:

sudo vim /etc/fstab

Add the following information:

<share> /mount/location virtiofs rw,_netdev 0 0

The file will mount on boot, but you can automatically mount with the following:

sudo mount -a

Reference: https://virtio-fs.gitlab.io/

A Simple Understanding of Docker :Analogy with Cargo Ship | by Akash  Agarwal | Medium
Docker Logos - Docker

Docker is a Platform as a Service (PaaS) product that leverages OS-Level Virtualization to deliver software packages referred to as “containers”. These containers are small VMs that run isolated from the host OS and provide software, services, libraries, and config files. The cool thing about Docker is that the containers are easy to setup and remove.

Like Virtual Machines there are a few concepts that containers have that are similar to a traditional bare metal computer:

  • CONTAINER: A runtime instance of the Docker image (similar to a VM)
  • IMAGE: Like a system image, these are preconfigured
  • VOLUME: A place to store persistent data since images are static
  • STACK: A cluster of containers that are managed together
  • NETWORKS:
  • ENV: Environmental variables are used to set the environment for commands, daemons, and processes.

Install Docker Debian (Ubuntu)

Setup the repo:

sudo apt-get update && sudo apt-get install ca-certificates curl gnupg lsb-release -y

Add docker GPG key:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Setup stable repo:

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Install docker:

sudo apt-get update && sudo apt-get install docker-ce docker-ce-cli containerd.io -y

Install Docker Compose

Install Docker Python Module

pip install docker

Connect to Docker in CLI

docker exec -it <container> /bin/bash

Watchtower (Update Containers Autimatically)

Watchtower

Watchtower is an application that will monitor your running Docker containers and watch for changes to the images that those containers were originally started from. If watchtower detects that an image has changed, it will automatically restart the container using the new image.

Watchtower is a containerized application that runs inside of Docker and monitors existing containers that are running and monitors for the latest image updates in the Docker hub. When Watchtower observes a new image, it pulls that image and restarts the container with that new image automatically

PULL the Watchtower image:

sudo docker pull containrrr/watchtower

DEPLOY Watchtower container:

docker run -d --name watchtower -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower

This will start a running Watchtower container.
Set Watchtower to RUN ONCE and close afterwards:

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --run-once

Volumes

CONTAINERHOST
/var/run/docker.sock/var/run/docker.sock

Environmental Variables

NAMEVALUE
WATCHTOWER_CLEANUPtrue
WATCHTOWER_INCLUDE_STOPPEDtrue
WATCHTOWER_POLL_INTERVAL3600
WATCHTOWER_REVIVE_STOPPEDtrue

Reference: https://hub.docker.com/r/containrrr/watchtower

Portainer Web UI (Docker)

Portainer is a Web User Interface for Docker.

Pull the image:

docker volume create portainer_data

Use Docker run to deploy the container:

docker run -d \
-p 8000:8000 \
-p 9000:9000 \
--name=portainer \
--restart=always \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /mnt/docker/images/staging/portainer/data:/data \
portainer/portainer-ce

You can access the portainer web interface over port 9000. You will need to initially setup by entering a username and password

Reference: https://docs.portainer.io/v/ce-2.9/start/install/server/docker/linux

Converting VMDK to Docker Image

Convert the WEMU image into a raw image:

sudo -img convert -f vmdk -O raw diskimage.vmdk image.img

After the Raw image is created, apply Guestfish:

guestfish -a image.img --ro
><fs> run

><fs> list-filesystems
/dev/sda1: ext4
/dev/VolGroup/lv_root: ext4
/dev/VolGroup/lv_swap: swap
><fs> mount /dev/VolGroup/lv_root /
><fs> tar-out / - | xz --best >>  myimage.xz
><fs> exit

Import the image into Docker:

cat mytry.xz | docker import - mydockerimage

Run the container:

docker run -it mydockerimage bash

Reference: https://blog.inslash.com/how-to-convert-vmdk-to-a-docker-image-be939745ed8a

Add a Docker Network with NO Internet Access

docker network create --subnet 172.19.0.0/16 no-internet

sudo iptables --insert DOCKER-USER -s 172.19.0.0/16 -j REJECT --reject-with icmp-port-unreachable

sudo iptables --insert DOCKER-USER -s 172.19.0.0/16 -m state --state RELATED,ESTABLISHED -j RETURN

Connect Docker Container to Host Network

Install Packages in Docker Container with APK

This is useful for docker containers that don’t use APT:

apk update && apk add vim

CNCF Branding | Kubernetes

I’ll add this eventually….. chill.

Configuring Docker Swarm as Container Orchestrator on Windows | by Aram  Koukia | Koukia

I’ll get to this too…..one day.

Webtop - Running Linux Desktop inside a Container in the Browser | by Hari  Prasad | featurepreneur | Medium

A webtop is a container that runs a virtual desktop environment in a web browser. Webstops are quick and easy to deploy and can be super useful for testing environments. Unlike a traditional Docker container, you would update the webtop like you would normally update any other Linux server.

One more consideration, webtops should NOT be exposed to the public and should instead lie safely behind a firewall.

docker pull linuxserver/webtop

Refer to the table below for different architectures:

ArchitectureTag
x86-64linuxserver/webtop:amd64-latest
arm64linuxserver/webtop:arm64v8-latest
armhflinuxserver/webtop:arm32v7-latest

Refer to the table below for different installs

DescriptionTag
XFCE Alpinelinuxserver/webtop:latest
XFCE Ubuntulinuxserver/webtop:ubuntu-xfce
XFCE Fedoralinuxserver/webtop:fedora-xfce
XFCE Archlinuxserver/webtop:arch-xfce
KDE Alpinelinuxserver/webtop:alpine-kde
KDE Ubuntulinuxserver/webtop:ubuntu-kde
KDE Fedoralinuxserver/webtop:fedora-kde
KDE Archlinuxserver/webtop:arch-kde
MATE Alpinelinuxserver/webtop:alpine-mate
MATE Ubuntulinuxserver/webtop:ubuntu-mate
MATE Fedoralinuxserver/webtop:fedora-mate
MATE Archlinuxserver/webtop:arch-mate
i3 Alpinelinuxserver/webtop:alpine-i3
i3 Ubuntulinuxserver/webtop:ubuntu-i3
i3 Fedoralinuxserver/webtop:fedora-i3
i3 Archlinuxserver/webtop:arch-i3
Openbox Alpinelinuxserver/webtop:alpine-openbox
Openbox Ubuntulinuxserver/webtop:ubuntu-openbox
Openbox Fedoralinuxserver/webtop:fedora-openbox
Openbox Archlinuxserver/webtop:arch-openbox
IceWM Alpinelinuxserver/webtop:alpine-icewm
IceWM Ubuntulinuxserver/webtop:ubuntu-icewm
IceWM Fedoralinuxserver/webtop:fedora-icewm
IceWM Archlinuxserver/webtop:arch-icewm

Use Docker run:

docker run -d \
  --name=webtop \
  --security-opt seccomp=unconfined `#optional` \
  -e PUID=1000 \
  -e PGID=1000 \
  -e SUBFOLDER=/ `#optional` \
  -e KEYBOARD=en-us-qwerty `#optional` \
  -p 3000:3000 \
  -v /docker/images/staging/webtop/config:/config \
  -v /var/run/docker.sock:/var/run/docker.sock `#optional` \
  --device /dev/dri:/dev/dri `#optional` \
  --shm-size="1gb" `#optional` \
  --restart unless-stopped \
  lscr.io/linuxserver/webtop
Kasm: A secure computing platform

Network Manager

Installing Network manager:

# Debian (Ubuntu)
sudo apt install NetworkManager

# Arch
sudo pacman -S NetworkManager

Connection Management

Check the status of Network Manager:

sudo systemctl status NetworkManager

List connection profiles:

# List connection profiles
nmcli connection show [option] {argument}

# Activate a connection
nmcli connection up [option] {argument}

#Deactivate a connection
nmcli connection down [option] {argument}

# Modify a connection
nmcli connection edit [option] {argument}

Device Management

Show device status:

nmcli device status

Show device

Enable Network Manager in Debian (Ubuntu)

sudo mv /usr/lib/NetworkManager/conf.d/10-globally-managed-devices.conf  /usr/lib/NetworkManager/conf.d/10-globally-managed-devices.conf_orig

sudo touch /usr/lib/NetworkManager/conf.d/10-globally-managed-devices.conf

sudo sed -i 's/managed\=false/managed\=true/g' /etc/NetworkManager/NetworkManager.conf

sudo systemctl restart NetworkManager

# You need to add the following to the yaml file in 'etc'netplan':
# renderer: NetworkManager

sudo sed -i '/^network:.*/a \ \ renderer: NetworkManager' /etc/netplan/50-cloud-init.yaml

# You can just purge the system from netplan

sudo apt update && sudo apt install ifupdown

sudo systemctl stop systemd-networkd \
sudo systemctl disable systemd-networkd \
sudo systemctl mask systemd-networkd

sudo apt --purge remove nplan netplan.io -y

Local Storage

List information about block devices:

lsblk -Mf

# useful list
lsblk -Mo NAME,SIZE,UUID,SERIAL,MOUNTPOINT

# alternative (lbslk is prefered)
sudo blkid 

List information about mounted block devices:

findmnt

List information for files in a directory:

ls -lah

# look recursively
ls -lahR
or
ls -lah *

List information for storage space on a file system:

df -h

List information for storage space for a directory

sudo du -hd1

You can alternatively use NCDU which is a better tool for quickly identifying and searching for disk space usage:

#Debian (Ubuntu)
sudo apt install ncdu

#Arch Linux
sudo pacman -S ncdu

Just type “ncdu” in the terminal and you’re done.

Disk Partitioning (fdisk)

List information about disk partitions:

sudo fdisk -l

Edit a disk partition

sudo fdisk /dev/<device>

Mounting Disks

You can mount disks manually using the “mount” command:

sudo mount /dev/<device> /mount/location

Unmount a disk:

sudo umount /mount/location

Mounting disks automatically with fstab (file systems tale):

sudo vim /etc/fstab

Enter the following information (you need the “UUID” and “Mount Location”):

UUID=<UUID> /mount/location auto defaults 0 0

The file will mount on boot, but you can automatically mount with the following:

sudo mount -a

Mounting exFAT

Windows filesystem “exFAT” is not natively recognized by Linux. Install the following utility to use exFAT filesystems:

sudo apt-get install exfat-fuse exfat-utils

Mounting Filesystems with NFS

Network File System allows the mounting of remote filesystems on a client from a remote server.

NFS Server

#Ubuntu
sudo apt update && sudo apt install nfs-kernel-server -y

#Arch Linux
sudo pacman -S nfs-utils

NFS Client

#Ubuntu
sudo apt update && sudo apt install nfs-common -y

#Arch Linux
sudo pacman -S nfs-utils

Remote Storage with SAMBA (SMB)

File:Logo Samba Software.svg - Wikimedia Commons

SAMBA Server

Verify if SAMBA is installed:

ls -l /etc/samba

To install SAMBA:

sudo apt-get update && sudo apt-get install samba -y

Set a password for smb user:

sudo smbpasswd -a <user_name>

Edit the smb conf file (If the smb.conf file does not exit, you can find it HERE):

sudo vim /etc/samba/smb.conf

Enter the following information based on your needs:

<share name>
path = /<location of the shared folder>
valid users = <username>
read only = no
guest ok = yes
guest only = yes
writable = yes
read only = yes
force user = <username>
force group = <groupname>
valid users = <username1>, <username2>, etc.

Mount SAMBA Shares to Client

List all available shares on a server:

smbclient -L //<host>

Install cifs-utils

sudo apt-get install cifs-utils

Manually mount the SAMBA share:

sudo mount -t cifs //<host>/<share> /mount/location

To automatically mount shares on boot, edit the fstab file:

sudo vim /etc/fstab

Enter the following information (you need the ):

//<host>/<share> /mount/location cifs credentials=/<filename>,iocharset=utf8,file_mode=0777,dir_mode=0777 0 0

You will need to make a password file and include it on the client with the following information:

user=USERNAME
password=PASSWORD
domain=SERVER

The file will mount on boot, but you can automatically mount with the following:

sudo mount -a

Mount Filesystems with SSHFS (SSH Filesystem)

SSH is a powerful networking tool for connecting to a host remotely over the command line interface. Here are some tips on how to use SSH to your advantage.

SSH usage

ssh <user>@<host>

Alternatively you can SSH over a specified port:

ssh -p <port> <user>@<host>

To specify a private key to use:

ssh -i </file/path> <user>@<host>

Generate an SSH key

ssh-keygen

Copy SSH keys to remote server for passwordless login:

ssh-copy-id <user>@<host>

Remove keys belonging to a user from the known_hosts file:

ssh-keygen -r <host>

This is very useful if you encounter this when you encounter this message when trying to login to a remote server (This is due to the host server being different than the one in the known_hosts file)

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.

SSH Tunneling (Port Forwarding)

Local Port Forwarding

Forward a port on a local SSH client to a remote SSH server that is forwarded to the destination:

ssh -L <lport>:<dest_host>:<dest_port> <user>@<rhost>

This is used for gaining access to a remote service.

ssh -L <lport>:<dest_host>:<dest_port> <lport>:<dest_host>:<dest_port> <user>@<rhost>

Remote Port Forwarding

Forward a port on a remote SSH server to a local SSH client that is forwarded to the destination:

ssh -R <rport>:<dest_host>:<dest_port> <user>@<rhost>

This is used for granting access to a remote user for a local service.

Dynamic Port Forwarding

Create a socket on the local SSH client

ssh -D <port> <user>@<host>

This is mostly used to tunnel web traffic (an alternative to using a VPN)

The -N option instructs not to execute a remote command and the -v is for verbosity:

ssh -NvD <port> <user>@<rhost>

Running Commands and Scripts on a Remote Host with SSH

This will allow you to execute one command or a series of commands using SSH. The command will login to the remote host, execute the command and terminate:

ssh -t <user>@<host> "<command> && <command>"

Here is a script that can be executed to run commands on multiple machines:

for s in <host1> <host2> <host3>
do
   ssh <user>@${s} <command>
done

You can even run a local bash script to a remote host:

ssh <user>@<host> 'bash -s' < /path/<script.sh>

The script for executing bash scripts on multiple hosts is similar to before:

for s in <host1> <host2> <host3>
do
   ssh <user>@${s} 'bash -s' < /path/<script.sh>
done

SSH Config File

SSH file should be located at ‘~/.ssh/config’. The file may not exist and in that case needs to be generated:

touch ~/.ssh/config

chmod 600 ~/.ssh/config

SSH config file format

Host <hostname>
    <SSH_OPTION> <value>

Use the following as an example:

Host sshserver
    HostName 192.168.1.100
    User 192.168.1.100
    Port 2222
    Compression yes
    IdentityFile ~/.ssh/keys/id_rsa

With this config file, you an automatically log into the server by using the following:

ssh sshserver

Transfer Files with SFTP (SSH File Transfer Protocol)

Use SFTP to transfer files to/from a remote server:

sftp <user>@<host>

These are the following common commands to use with SFTP

ls    (list files and directories on remote host)
lls   (list files and directories on local host)
cd    (change directories on remote host)
lcd   (change directories on local host)
pwd   (list present working directory on remote host)
lpwd  (list present working directory on local host)
get   (retrieve file from remote host)
mget  (retrieve multiple files from remote host)
put   (place file on remote host)
mput  (pace multiple files on remote host)

Transfer Files with SCP (Secure Copy Protocol)

Copy remote file to a local system:

scp <file> <user>@<host>:/remote/directory

Copy a local file to a remote system:

scp <user>@<host>:<file> /local/directory

Copy files between two remote hosts:

scp <user>@<host>:/remote/directory <user>@<host>:/remote/directory

Use these options:

-P   (Specify SSH port)
-r   (Recursive)
-C   (Compress data)

Mount Filesystems with SSHFS (SSH Filesystem)

Install SSHFS:

# Debian (Ubuntu)
sudo apt install sshfs

# Arch
sudo pacman -S sshfs

Mount a remote share with SSHFS:

sudo sshfs <user>@<host>:/remote/dir /mount/location <options>

To unmount:

sudo umount /mount/location

Mount automatically by adding to fstab:

sudo vim /etc/fstab

Add the following to ftsab:

<user>@<host>:/remote/dir /mount/location fuse.sshfs defaults 0 0

To login without a password, make sure the SSH key is stored on the remote server. You can do with with “ssh-copy-id”. The private SSH also needs to be in the root directory “/root/.ssh”.

With the default entry, you will have to use the root user in order to cd into that directory or access the files. To allow users to access directories and files, use the -o option or “allow_others”.

sshfs -o allow_other

#or mount to /etc/fstab

<user>@<host>:/remote/dir /mount/location fuse.sshfs defaults,allow_other 0 0

Install OpenSSH Server (Docker)

The OpenSSH server in a container will allow you to give SSH permission for a client to a resource without giving them full permission to the entire server.

Pull the “OpenSSH Server” container image from Docker Hub

docker pull linuxserver/openssh-server

Use Docker run:

docker run -d \
--name=openssh-server \
-e PUID=1000 \
-e PGID=1000 \
-e SUDO_ACCESS=false \
-e USER_PASSWORD_FILE=/path/to/file `#optional` \
-p 2222:2222 \
-v /path/to/config:/config \
--restart unless-stopped \
lscr.io/linuxserver/openssh-server

Reference: https://hub.docker.com/r/linuxserver/openssh-server

Install Endlessh (Docker)

Endlessh is an SSH tarpit that very slowly sends an endless, random SSH banner. It keeps SSH clients locked up for hours or even days at a time. The purpose is to put your real SSH server on another port and then let the script kiddies get stuck in this tarpit instead of bothering a real server.

WARNING
Before setting up Endlessh, make sure you change the default ssh port on the server from port 22 for a different port.

Pull the “Endlessh” container image from Docker Hub:

docker pull linuxserver/endlessh

Use Docker run:

docker run -d \
  --name=endlessh \
  -e PUID=1000 \
  -e PGID=1000 \
  -p 22:2222 \
  -MSDELAY=7000 \
  -MAXLINES=32 \
  -MAXCLIENTS=4096 \
  -LOGFILE=true \
  -v /mnt/docker/images/staging/endlessh/config:/config \
  --restart unless-stopped \
  lscr.io/linuxserver/endlessh

Reference: https://github.com/skeeto/endlessh
Reference: https://hub.docker.com/r/linuxserver/endlessh

VPN over SSH with sshuttle

Sshuttle is a VPN solution for SSH without the complexities of port forwarding.

#Install Debian (Ubuntu)
sudo apt get update && apt get install sshuttle

#Install Arch
sudo pacman -S sshuttle

If you choose, you can use Git to install.

git clone https://github.com/sshuttle/sshuttle.git
cd sshuttle
sudo ./setup.py install

Reference: https://github.com/sshuttle/sshuttle

Surfshark (With Glutun Docker Container)

Media Center - Surfshark

Surfshark is an awesome public VPN that can be used to anonymize your traffic and surf the web without exposing a lot of your personal information. Using a public VPN service provider is not a magic one-stop solution for internet security, but it is an important layer.

Most VPN providers limit the number of devices you can have connected under your account, but Surfshark does not. This is important because I like to configure Surfshark to run on a lot of my systems (and I have a lot) and removing that cap that most other providers implement gives me a lot of freedom.

I deploy Surfshark in a Docker Container called “Glutun”.

sudo docker run -d \
--name <name> \
--cap-add=NET_ADMIN \
-e VPNSP="surfshark" \
-e OPENVPN_USER=<uname> \
-e OPENVPN_PASSWORD=<passwd> \
-v /mnt/docker/staging/gluetun:/gluetun \
qmcgaw/gluetun

There are a few variables that you are going to want to change here and I will list them below:

Environmental Variables

NAMEVALUE
VPNSPsurfshark
COUNTRYRefer to “Server Locations” below
CITYRefer to “Server Locations” below
OPENVPN_USERSurfshark username
OPENVPN_PASSWORDSurfshark password

WireGuard (Docker Container)

File:Logo of WireGuard.svg - Wikimedia Commons

WireGuard Server

Wireguard is installed using a docker container image provided by linuxserver.io

sudo docker run -d \
  --name=wireguard \
  --cap-add=NET_ADMIN \
  --cap-add=SYS_MODULE \
  -e PUID=1000 \
  -e PGID=1000 \
  -e SERVERURL=host \
  -e SERVERPORT=port \
  -e PEERS=1 \
  -e PEERDNS=auto \
  -p port:port/udp \
  -v /mnt/docker/staging/wireguard/config:/config \
  -v /lib/modules:/lib/modules \
  --sysctl="net.ipv4.conf.all.src_valid_mark=1" \
  --restart unless-stopped \
  linuxserver/wireguard

Environmental Variables

NAMEVALUE
SERVERURLIP address or URL location of the VPN server
SERVERPORTExternal port
PEERSNumber of simultaneous connected peers
PEERDNSauto

Volumes

CONTAINERHOST
/file/path/config
/lib/modules/lib/modules

WireGuard Client

VPN over SSH with sshuttle

Sshuttle is a VPN solution for SSH without the complexities of port forwarding.

#Install Debian (Ubuntu)
sudo apt get update && apt get install sshuttle

#Install Arch
sudo pacman -S sshuttle

If you choose, you can use Git to install.

git clone https://github.com/sshuttle/sshuttle.git
cd sshuttle
sudo ./setup.py install

Reference: https://github.com/sshuttle/sshuttle

Build a Website with WordPress (Docker)

WordPress is the most popular CMS (Content Management System) in the world. Its easy to deploy a WordPress container and start building your website today. Using Docker is much better than setting up a traditional LAMP server.

Install Portainer Web UI for Docker

Portainer is a Web User Interface for Docker.

Pull the image:

docker volume create portainer_data

Use Docker run to deploy the container:

docker run -d \
-p 8000:8000 \
-p 9000:9000 \
--name=portainer \
--restart=always \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /mnt/docker/staging/portainer/data:/data \
portainer/portainer-ce

You can access the portainer web interface over port 9000. You will need to initially setup by entering a username and password

Reference: https://docs.portainer.io/v/ce-2.9/start/install/server/docker/linux

Install NGINX Reverse Proxy Manager

Nginx Proxy Manager

A reverse proxy manager will

docker run -d \
--name=nginx-proxy-manager
-p 8080:80 \
-p 81:81 \
-p 4433:443 \
-v /mnt/docker/staging/nginx_proxy_manager/data:/data \
-v /mnt/docker/staging/nginx_proxy_manager/etc/letsencrypt:/etc/letsencrypt \
--restart unless-stopped \
jc21/nginx-proxy-manager:latest

After creating the container, login to the web interface by typing in the IP address with the port 81.

The default login credentials are:
USER: [email protected]
PASSWD: changeme

Once you successfully login, you will be asked to change these.

Reference: https://nginxproxymanager.com/guide/

Deploy WordPress Stack (Docker)

Login to Portainer web interface and navigate to “App Templates”

This image has an empty alt attribute; its file name is image-14-1024x287.png

Name the stack and pick a password for the database.

This image has an empty alt attribute; its file name is image-13-1024x565.png

Determine which external port the WordPress container is mapped to (for example you might see 48312:80). In this case, you would navigate to the IP address and the port number 48213.

This image has an empty alt attribute; its file name is image-18.png

Complete the login

Increase WordPress MAX File Upload Size

Edit .htaccess:

php_value upload_max_filesize 64M
php_value post_max_size 64M
php_value max_execution_time 300
php_value max_input_time 300

Change the user permissions on the file otherwise it may revert back:

sudo chmod 600 .htaccess

Searching

Searching for a file or directory name (use the “find” command):

sudo find . -name sample.txt

#search in a specific directory
sudo find /home -name sample.txt

#use astrisk as a wildcard character
sudo find . -name *sample*

#ignore case sensitivity using "iname"
sudo find . -iname sample.txt

#search for a directory name with "-type d"
sudo find . -type d -name sample

GREP

Speeding up grep searches

Use LC_ALL=C:

LC_ALL=C grep example file.txt

Use fgrep if you are searching for a fixed string instead of a regular expression

fgrep example file.txt

Use parallel

Search for an IP address for an IP address

grep -oE "(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)"
Bash Logo Media Assets - Download Bash shell logo - Bourne-again shell logo

Create a file and make it executable

touch <filename>
sudo chmod +x <filename>

TO add the shebang, we need to determine the interpreter:

which bash

Add that to the top of the file

#!/bin/bash

Strings

Variables

Global Variable

Global variables can be used anywhere throughout the script

#!/bin/bash
VAR="global string"
echo $VAR

global string

Local Variable

Local variables can only be called to bash functions

#!/bin/bash
local VAR="local variable"
echo $VAR

local variable

User Input

Arguments

Arrays

Loops

For

While

Until

If / Elif / Else / Fi Statement

Adding this for now (since Yes/no query in BASH):

while true; do
   read -p "statement" yn
   case ${yn:0:1} in
      [Yy]* )
         if 
            <commands>
         else
            <commands>
         fi
         break;;
      [Nn]* )
         exit;;
      * ) echo "Please answer either Y/y or N/n";;
   esac
done

The Python Logo | Python Software Foundation
Automate the Boring Stuff with Python: Practical Programming for Total  Beginners 1, Sweigart, Al, eBook - Amazon.com

Install Pip

Install Python3 Pip for Debian (Ubuntu):

sudo apt update && sudo apt install python3-pip -y

Verify version number:

pip3 --version

Ansible is Simple IT Automation

Install Ansible on Debian (Ubuntu)

sudo apt update \
sudo apt install software-properties-common -y \
sudo add-apt-repository --yes --update ppa:ansible/ansible \
sudo apt install ansible -y

Install Semaphore Web Interface (Docker)

Install Ansible Semaphore on Linux | Snap Store

This container is installed using a yaml file with docker-compose

version: '2'

services:

  mysql:
    ports:
      - 3306:3306
    image: mysql:5.6
    hostname: mysql
    environment:
      MYSQL_RANDOM_ROOT_PASSWORD: 'yes'
      MYSQL_DATABASE: semaphore
      MYSQL_USER: <CHANGE>
      MYSQL_PASSWORD:<CHANGE>

  semaphore:
    ports:
      - 3000:3000
    image: ansiblesemaphore/semaphore:latest
    environment:
      SEMAPHORE_DB_USER: <CHANGE>
      SEMAPHORE_DB_PASS: <CHANGE>
      SEMAPHORE_DB_HOST: mysql
      SEMAPHORE_DB_PORT: 3306
      SEMAPHORE_DB: semaphore
      SEMAPHORE_PLAYBOOK_PATH: /tmp/semaphore/
      SEMAPHORE_ADMIN_PASSWORD: user
      SEMAPHORE_ADMIN_NAME: user
      SEMAPHORE_ADMIN_EMAIL: [email protected]
      SEMAPHORE_ADMIN: admin
      SEMAPHORE_ACCESS_KEY_ENCRYPTION: <CHANGE>
    depends_on:
      - mysql

You need to replace the following values:

  • MYSQL_USER
  • MYSQL_DATABASE
  • SEMAPHORE_DB_USER
  • SEMAPHORE_DB_PASS
  • SEMAPHORE_ACCESS_KEY_ENCRYPTION

Generate Access Key Encryption with the following command:

head -c32 /dev/urandom | base64

You can access the web interface over port 3000.

Reference: https://docs.ansible-semaphore.com/administration-guide/installation#docker

Install AWX Web Interface (Docker)

Reference: https://github.com/ansible/awx
Reference: https://debugthis.dev/posts/2019/10/setting-up-ansible-awx-using-a-docker-environment-part-1-the-ansible-approach/

Install the ultimate self-hosted media server with Plex or Jellyfin for FREE.
Host all of your photos and video files in a convenient, accessible and secure location from the internet.

These are ALL docker containers and therefore require Docker to be installed. Additionally, some of these steps include the use of Portainer as well as an NGINX Reverse Proxy (which can both be setup as Docker containers)

Install Portainer Web UI for Docker

Portainer is a Web User Interface for Docker.

Pull the image:

docker volume create portainer_data

Use Docker run to deploy the container:

docker run -d \
-p 8000:8000 \
-p 9000:9000 \
--name=portainer \
--restart=always \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /mnt/docker/staging/portainer/data:/data \
portainer/portainer-ce

You can access the portainer web interface over port 9000. You will need to initially setup by entering a username and password

Reference: https://docs.portainer.io/v/ce-2.9/start/install/server/docker/linux

Install NGINX Reverse Proxy Manager

Nginx Proxy Manager

A reverse proxy manager will

docker run -d \
--name=nginx-proxy-manager
-p 8080:80 \
-p 81:81 \
-p 4433:443 \
-v /file/path:/data \
-v /file/path:/etc/letsencrypt \
--restart unless-stopped \
jc21/nginx-proxy-manager:latest

After creating the container, login to the web interface by typing in the IP address with the port 81.

The default login credentials are:
USER: [email protected]
PASSWD: changeme

Once you successfully login, you will be asked to change these.

Reference: https://nginxproxymanager.com/guide/

Install Plex Media Server (Docker)

Install Portainer Web UI for Docker

Portainer is a Web User Interface for Docker.

Pull the image:

docker volume create portainer_data

Use Docker run to deploy the container:

docker run -d \
-p 8000:8000 \
-p 9000:9000 \
--name=portainer \
--restart=always \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /mnt/docker/staging/portainer/data:/data \
portainer/portainer-ce

You can access the portainer web interface over port 9000. You will need to initially setup by entering a username and password

Reference: https://docs.portainer.io/v/ce-2.9/start/install/server/docker/linux

Install NGINX Reverse Proxy Manager

Nginx Proxy Manager

A reverse proxy manager will

docker run -d \
--name=nginx-proxy-manager
-p 8080:80 \
-p 81:81 \
-p 4433:443 \
-v /file/path:/data \
-v /file/path:/etc/letsencrypt \
--restart unless-stopped \
jc21/nginx-proxy-manager:latest

After creating the container, login to the web interface by typing in the IP address with the port 81.

The default login credentials are:
USER: [email protected]
PASSWD: changeme

Once you successfully login, you will be asked to change these.

Reference: https://nginxproxymanager.com/guide/

Hosting with Cloudflare

host services from anywhere using Cloudflare DNS

hostnamectl

timedatectl

Honey Uhd 4k Wallpaper - Honey Wallpaper 4k (#3270407) - HD Wallpaper &  Backgrounds Download

Honeypots

Installing DShield Honeypot

Update the system and install Git

sudo apt update && sudo apt install -y git

Clone the DShield Git repository

git clone https://github.com/DShield-ISC/dshield.git

Run the install script

sudo ./dshield/bin/install.sh

Select YES

Select AUTOMATIC and OK

You need to create a dshield account at https://dshield.org/login.html

Get the API key and enter the information

Select OK

Select the default interface

Enter the network information

Confirm by clicking OK

Enter IPs to ignore

Confirm by clicking OK

Next enter IPs and ports to disable

Confirm by selecting OK

Enter information to create the SSL certificate

Confirm by clicking YES

Reboot the machine and login using the new SSH port (12222)

sudo reboot

Reference: https://securitytrails.com/blog/top-honeypots

Building a Website

Building a Webserver with NGINX (Docker)

HGINX can act as the following:

  • webserver
  • reverse proxy
  • load balancer
  • mail proxy
  • http cache

Reverse Proxy Manager with NGINX (Docker)

Nginx Proxy Manager

A reverse proxy manager will

docker run -d \
--name=nginx-proxy-manager \
-p 8080:80 \
-p 81:81 \
-p 4433:443 \
-v /mnt/docker/staging/nginx_proxy_manager/data:/data \
-v /mnt/docker/staging/nginx_proxy_manager/etc/letsencrypt:/etc/letsencrypt \
--restart unless-stopped \
jc21/nginx-proxy-manager:latest

After creating the container, login to the web interface by typing in the IP address with the port 81.

The default login credentials are:
USER: [email protected]
PASSWD: changeme

Once you successfully login, you will be asked to change these.

Reference: https://nginxproxymanager.com/guide/
Reference: https://hub.docker.com/r/jc21/nginx-proxy-manager

Reference: https://hub.docker.com/_/nginx
Reference: docker pull linuxserver/nginx

Building a Webserver with Apache (Docker)

Building a Website with WordPress (Docker)

Traefik (Docker)

Continuous Integration / Continuous Delivery with Jenkins (Docker)

File:Jenkins logo.svg - Wikimedia Commons

Pull the image:

docker pull jenkins/jenkins

Run the image with this command:

docker run -p 8080:8080 -p 50000:50000 --restart always -v jenkins_home:/var/jenkins_home jenkins/jenkins:lts-jdk11

This is useful for taking items separated by commas and turning them into a list of items, sorting and removing duplicate entries:

tr ' ' '\n' | tr -d , | sort -V | uniq

I used this to get a list of filenames based on date

ls -lat | grep -i <name> | grep -i <date> | cut -f2- -d: | cut -c 4-

Scroll to Top