<%- include('includes/header.ejs', {
    active_page: 'home'
}) %>
<h1>Infrastructure</h1>
<section id="table-of-contents">
    <h2>Table of Contents</h2>
    <ul>
        <li>
            <a href="#setup">Setup</a>
            <ul>
                <li>
                    <a href="#bare-metal">Bare Metal</a>
                </li>
                <li>
                    <a href="#drives-and-raid">Drives and RAID</a>
                </li>
                <li>
                    <a href="#server-housing">Server Housing</a>
                </li>
                <li>
                    <a href="#network-infrastructure">Network Infrastructure</a>
                </li>
                <li>
                    <a href="#power">Power (UPS + Power Management Unit)</a>
                </li>
                <li>
                    <a href="#monitoring-and-remote-control">Monitoring and Remote Control (IDRAC)</a>
                </li>
                <li>
                    <a href="#operating-system">Operating System (Proxmox)</a>
                </li>
                <li>
                    <a href="#kubernetes">Kubernetes</a>
                </li>
                <li>
                    <a href="#kubernetes-persistent-storage">Kubernetes Persistent Storage (Ceph/Rook)</a>
                </li>
                <li>
                    <a href="#private-container-registry">Private Container Registry (Harbor)</a>
                </li>
                <li>
                    <a href="#kubernetes-relational-database-cluster">Kubernetes Relational Database Cluster (CloudNativePG)</a>
                </li>
                <li>
                    <a href="#private-npm-registry">Private NPM Registry (Verdaccio)</a>
                </li>
                <li>
                    <a href="#cluster-and-service-monitoring">Cluster and Service Monitoring (Prometheus/Grafana)</a>
                </li>
                <li>
                    <a href="#web-anayltics">Web Analytics (Matomo)</a>
                </li>
                <li>
                    <a href="#cluster-ingress">Cluster Ingress (Nginx Ingress + MetalLB)</a>
                </li>
                <li>
                    <a href="#local-network-dns">Local Network DNS (Bind)</a>
                </li>
                <li>
                    <a href="#dhcp-and-filtering">DHCP and Filtering (PiHole)</a>
                </li>
                <li>
                    <a href="#public-reverse-proxy">Public Reverse Proxy (Nginx Proxy Manager)</a>
                </li>
                <li>
                    <a href="#public-service-monitoring">Public Service Monitoring (Uptime Kuma)</a>
                </li>
                <li>
                    <a href="#dynamic-dns">Dynamic DNS (No-IP)</a>
                </li>
                <li>
                    <a href="#public-dns">Public DNS (GoDaddy)</a>
                </li>
                <li>
                    <a href="#email">Email (Mailcow + M365)</a>
                </li>
                <li>
                    <a href="#alternate-port-smtp">Alternate Port SMTP (No-IP)</a>
                </li>
                <li>
                    <a href="#newsletter-and-transactional-emails">Newsletter and Transactional Emails (Listmonk)</a>
                </li>
                <li>
                    <a href="#social-media-management">Social Media Management (Mixpost)</a>
                </li>
                <li>
                    <a href="#phone">Phone (FreePBX)</a>
                </li>
                <li>
                    <a href="#groupware">Groupware (Nextcloud All-In-One)</a>
                </li>
                <li>
                    <a href="#homepage">Internal Services Dashboard (Homepage)</a>
                </li>
            </ul>
        </li>
        <li>
            <a href="#costs">Costs</a>
        </li>
        <li>
            <a href="#quick-links">Quick Links</a>
        </li>
    </ul>
</section>
<hr>
<section id="setup">
    <h2>Setup</h2>
    <div id="bare-metal">
        <h3>Bare Metal</h3>
        <p>
            We run almost all our workloads on our owned and operated "bare metal" servers.
            In particular, we bought 2 used servers:
            <ol>
                <li>Dell PowerEdge R730</li> 
                <li>Dell PowerEdge R730xd</li>
            </ol>
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="drives-and-raid">
        <h3>Drives and RAID</h3>
        <p>
            The used Dell PowerEdge R730 we bought came with it's 8 drive bays full with 2 TB hard drives.
            We run this using RAID 5, which means that we have 14 TB of storage and can have a single drive fail without losing any data.
        </p>
        <p>
            The Dell PowerEdge R730xd we bought unfortunately came with no drives (not having experience with buying servers and the difference between "drives" and "drive bays").
            So, we bought 4 1TB Samsung SSDs to fill in 4 of it's 24 drive bays. Which we also learned we had to buy caddys for (and did).
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="server-housing">
        <h3>Server Housing</h3>
        <p>
            We house our servers on rails inside a small 12U Sysracks server cabinet in our office.
            This cabinet came with a built in fan, a lockable front door (lockable but removable side panels) and a power management unit.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="network-infrastructure">
        <h3>Network Infrastructure</h3>
        <p>
            We pay our ISP for a 1.5 Gbps connection, which is run through our ISP provided modem/router (which is in Bridge Mode). 
            The ISP modem/router is then run to a TP-Link adapter (because we have a Powerline network connection to our office using another TP-Link adapter)
            We have to do this, this way because we lack the physical ability to run cabling through the physical structure.
            The TP-Link adapter in our office is then run to a Netgear Nighthawk router which creates vnets within our office. 
            Which, for the purposes of this, is feed to a Netgear 1Gbps 8 port switch inside of our server cabinet. 
            Which finally runs from the switch directly to our servers.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="power">
        <h3>Power (UPS + Power Management Unit)</h3>
        <p>
            We have a Tripplite 1500VA UPS that we use to power our servers for a short time in the event of a power outage.
            We have it plugged into the wall and then have our servers having one power supply plugged into it (the servers have two power supplies).
            We have the other power supply plugged into the power management unit in our server cabinet. Which the power management unit is plugged into the wall.
            We choose to set it up this way because we felt it was the architecture that best fit our risk profile. 
            That is, we attempt to ensure that our servers would stay up (at least for a time) in the event of a power outage or UPS failure.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="monitoring-and-remote-control">
        <h3>Monitoring and Remote Control (IDRAC)</h3>
        <p>
            Amongst other reasons we choose to buy Dell servers because we wanted access via Integrated Dell Remote Access Controller (IDRAC).
            We use IDRAC to monitor and control our servers on a hardware level remotely.
            Though, we do have access to many of the same features/metrics through software having IDRAC provides us a piece of mind backup just in case.
            We have it set up so that we can ONLY access the IDRAC interface from our office network or VPN.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="operating-system">
        <h3>Operating System (Proxmox)</h3>
        <p>
            We run <a href="https://www.proxmox.com/proxmox-virtual-environment/overview">Proxmox Virtual Environment (PVE)</a> which is based on Debian Linux on both of our servers.
            This allows us to easily run and manage virtual workflows to maximize our hardware utilization.
            We also have them connected as a cluster within the PVE interface so that we can: 
            <ul>
                <li>live migrate certain workloads between the two servers</li>
                <li>Have an active-backup setup for certain workloads</li>
                <li>Etc...</li>
            </ul>
            Though, it's worth noting this <strong>ISN'T</strong> a high availability cluster because you require at least a third server to do that.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="kubernetes">
        <h3>Kubernetes</h3>
        <p>
            We run a <a href="https://kubernetes.io/">Kubernetes</a> cluster on top of our Proxmox cluster.
            We use this to run many (most) of our workloads.
            In particular, any workloads that can be containerized we try to (and/or aspire to) run on our Kubernetes cluster.
        </p>
        <p>
            At time of writing, we run one control node and four worker nodes.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="kubernetes-persistent-storage">
        <h3>Kubernetes Persistent Storage (Ceph/Rook)</h3>
        <p>
            We run a <a href="https://ceph.io/">Ceph</a> cluster on top of Kubernetes using <a href="https://rook.io/">Rook</a>.
            We use this to store our persistent data.
            In particular, this includes storage for databases and other data that needs to be stored more permanently (across pod restarts/deployments).
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="private-container-registry">
        <h3>Private Container Registry (Harbor)</h3>
        <p>
            We run a private container registry on top of Kubernetes using <a href="https://goharbor.io/">Harbor</a>.
            We use this to store our container images.
            In particular, this includes images for our workloads that we deploy to our Kubernetes cluster.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="kubernetes-relational-database-cluster">
        <h3>Kubernetes Relational Database Cluster (CloudNativePG)</h3>
        <p>
            We run a PostgreSQL cluster on top of Kubernetes using <a href="https://cloudnative-pg.io/">CloudNativePG</a>.
            We use this to store our relational data.
            In particular, this provides us another level of redundancy and reliability for our data.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="private-npm-registry">
        <h3>Private NPM Registry (Verdaccio)</h3>
        <p>
            We run a private NPM registry on top of Kubernetes using <a href="https://verdaccio.org/">Verdaccio</a>.
            We use this to store our NPM packages.
            In particular, this includes packages we tend to use with Node and other JavaScript based projects.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <!-- <a href="https://github.com/pypiserver/pypiserver">pypiserver</a> - Private PyPI registry? -->
    <div id="cluster-and-service-monitoring">
        <h3>Cluster and Service Monitoring (Prometheus/Grafana)</h3>
        <p>
            We run a <a href="https://prometheus.io/">Prometheus</a>/<a href="https://grafana.com/">Grafana</a> stack on top of Kubernetes.
            We use this to monitor our cluster and services.
            In particular, we use this to monitor the health of our cluster and services.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="web-anayltics">
        <h3>Web Analytics (Matomo)</h3>
        <p>
            We run a <a href="https://matomo.org/">Matomo</a> server on top of Kubernetes.
            We use this to track web traffic and usage statistics of our websites.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="cluster-ingress">
        <h3>Cluster Ingress (Nginx Ingress + MetalLB)</h3>
        <p>
            We run an <a href="https://github.com/kubernetes/ingress-nginx">Nginx Ingress controller</a> alongside <a href="https://metallb.universe.tf/">MetalLB</a> on top of Kubernetes.
            We use this to route traffic to our services deployed on our Kubernetes cluster.
            The NGNIX Ingress controller is used to route traffic to our services and MetalLB is used, in our case, to obtain a network accessible IP address for the NGINX Ingress controller.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="local-network-dns">
        <h3>Local Network DNS (Bind)</h3>
        <p>
            We run two <a href="https://www.isc.org/bind/">Bind</a> DNS servers (a primary and secondary) outside our Kubernetes cluster.
            We use this to resolve DNS queries for our local corporate network.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="dhcp-and-filtering">
        <h3>DHCP and Filtering (Pi-hole)</h3>
        <p>
            We run a <a href="https://pi-hole.net/">Pi-hole</a> server outside our Kubernetes cluster.
            We use this to provide DHCP and filter DNS queries for our local corporate network.
            It's worth noting that we generally prefer to use static DHCP leases over static IP addresses as it provides us more flexibility and easy oversight.
            That is, we can go to Pi-hole and see a single table with all the static leases rather than relying on a spreadsheet or other documentation to keep track of assigned static IPs
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="public-reverse-proxy">
        <h3>Public Reverse Proxy (Nginx Proxy Manager)</h3>
        <p>
            We run a <a href="https://nginxproxymanager.com/">Nginx Proxy Manager (NPM)</a> server outside of our Kubernetes cluster on a separate server/computer from the rest of our infrastructure.
            We use this to have more granular control of how (particularly public) traffic is routed within our network/to our services.
            Moreover, this usually handles TLS/SSL termination for our public services and getting and maintaining certificates through Let's Encrypt.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="public-service-monitoring">
        <h3>Public Service Monitoring (Uptime Kuma)</h3>
        <p>
            We run an <a href="https://uptime.kuma.pet/">Uptime Kuma</a> server outside of our Kubernetes cluster on a separate server/computer from the rest of our infrastructure.
            We use this to monitor the uptime of/access to our public services.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="dynamic-dns">
        <h3>Dynamic DNS (No-IP)</h3>
        <p>
            We use <a href="https://www.noip.com/">No-IP</a> to provide us with a Dynamic DNS (DDNS) service for a small cost.
            That is, because our ISP wouldn't provide us a static IP address with the speed we wanted we use a DDNS service to have a name that resolves to our dynamic IP address.
            What this means practically, is we have a piece of software (provided by No-IP) running on one of our computers/servers that updates the No-IP record if our public IP address ever changes. 
            We then use this in CNAME records for our public domain name so that our services can be publicly available without having to worry if our IP address ever changes.
            Though for the A records at the root of our domains we had to develop a script that did similar/the same as the No-IP software.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="public-dns">
        <h3>Public DNS (GoDaddy)</h3>
        <p>
            We use <a href="https://www.godaddy.com/">GoDaddy</a> to provide us with public DNS services for a cost.
            That is, we use GoDaddy to manage our public domain names and the DNS records associated with them.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="email">
        <h3>Email (Mailcow + M365)</h3>
        <p>
            We use a weird mix of services for email.
            Largely this is split between our self-hosted <a href="https://mailcow.email/">Mailcow</a> server and Microsoft 365.
            Our Mailcow server is used is used for internal email and access for our services.
            While Microsoft 365 is mostly used for external email.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="alternate-port-smtp">
        <h3>Alternate Port SMTP (No-IP)</h3>
        <p>
            We use <a href="https://www.noip.com/managed-mail#self-hosted">No-IP's alternate port SMTP service</a> for a small cost.
            That is, because our ISP blocks outbound traffic on port 25 we use a service that provides us with an alternate port to send email.
            We then configure this in our Mailcow server to send email out to the internet.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="newsletter-and-transactional-emails">
        <h3>Newsletter and Transactional Emails (Listmonk)</h3>
        <p>
            We use <a href="https://listmonk.app/">Listmonk</a> to send out our newsletters and transactional emails.
            Listmonk is a self-hosted solution that we host on our Kubernetes cluster.
            This connects with our Mailcow server to send out emails.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="social-media-management">
        <h3>Social Media Management (Mixpost)</h3>
        <p>
            We use <a href="https://mixpost.app/">Mixpost</a> to manage our social media accounts.
            Mixpost is a self-hosted solution that we host on our Kubernetes cluster.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="phone">
        <h3>Phone (FreePBX)</h3>
        <p>
            We use <a href="https://www.freepbx.org/">FreePBX</a> to provide us with phone services.
            This is a self-hosted solution that we host outside our Kubernetes cluster.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="groupware">
        <h3>Groupware (Nextcloud All-In-One)</h3>
        <p>
            We use <a href="https://github.com/nextcloud/all-in-one">Nextcloud All-In-One (AIO)</a> to provide us with groupware services.
            This is a self-hosted solution that we host outside our Kubernetes cluster (though aspirational would like to move onto Kubernetes).
            This provides us with file storage, calendar, contacts, and other groupware services.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
    <div id="homepage">
        <h3>Internal Services Dashboard (Homepage)</h3>
        <p>
            We use <a href="https://gethomepage.dev/">Hmmepage</a> to provide a custom built dashboard of our services for internal users.
            This is a simple HTML page that we host on our Kubernetes cluster.
            It provides us with a quick links and an overview of the status of our services.
        </p>
        <div>
            <a href="#table-of-contents">Go to Table of Contents</a>
        </div>
    </div>
</section>
<hr>
<section id="costs">
    <h2>Costs</h2>
    <table>
        <thead>
            <tr>
                <th>Item</th>
                <th>Cost</th>
                <th>Notes</th>
            </tr>
        </thead>
        <tbody>
            <tr>
                <td>Dell PowerEdge R730</td>
                <td>$1,140.14</td>
                <td>Used server from Amazon.ca</td>
            </tr>
            <tr>
                <td>Dell PowerEdge R730xd</td>
                <td>$1,180.52</td>
                <td>Used server from Amazon.ca</td>
            </tr>
            <tr>
                <td>Dell Server Rack Rails (&times; 2)</td>
                <td>$288.96</td>
                <td>Used from Amazon.ca</td>
            </tr>
            <tr>
                <td>12 pack 2.5&apos;&apos; Drive Caddy (&times; 2)</td>
                <td>$239.40</td>
                <td>From Amazon.ca</td>
            </tr>
            <tr>
                <td>1TB Samsung SSDs (&times; 4)</td>
                <td>$582.36</td>
                <td>From Amazon.ca</td>
            </tr>
            <tr>
                <td>12U Sysracks server cabinet</td>
                <td>$547.70</td>
                <td>From Amazon.ca</td>
            </tr>
            <tr>
                <td>TP-Link adapter</td>
                <td>$291.18</td>
                <td>From Amazon.ca</td>
            </tr>
            <tr>
                <td>Netgear Nighthawk router</td>
                <td>$146.44</td>
                <td>Used from individual</td>
            </tr>
            <tr>
                <td>Netgear 1Gbps 8 port switch</td>
                <td>$58.98</td>
                <td>Used from individual</td>
            </tr>
            <tr>
                <td>Tripplite 1500VA UPS</td>
                <td>$517.87</td>
                <td>From Amazon.ca</td>
            </tr>
            <tr>
                <td>Power</td>
                <td>$597.00/year</td>
                <td>Approximate</td>
            </tr>
            <tr>
                <td>Internet (from ISP)</td>
                <td>$1,464.96/month</td>
                <td>1.5Gbps speed</td>
            </tr>
            <tr>
                <td>DDNS (from No-IP)</td>
                <td>$99.99/year</td>
                <td>Dynamic DNS service</td>
            </tr>
            <tr>
                <td>Public DNS (from GoDaddy)</td>
                <td>$401.39/year</td>
                <td>DNS (for multiple domains) + M365</td>
            </tr>
            <tr>
                <td>Alternate Port SMTP (from No-IP)</td>
                <td>$39.99/year</td>
                <td>Alternate port SMTP service</td>
            </tr>
        </tbody>
        <tfoot>
            <tr>
                <td>Fixed One Time/Capital</td>
                <td>$4,993.63</td>
            </tr>
            <tr>
                <td>Variable (Yearly Total)</td>
                <td>$2,603.33</td>
                <td>Or ~$216.95 per month</td>
            </tr>
        </tfoot>
    </table>
    <div>
        <a href="#table-of-contents">Go to Table of Contents</a>
    </div>
</section>
<hr>
<section id="quick-links">
    <ul>
        <li>
            <a href="https://www.proxmox.com/proxmox-virtual-environment/overview">Proxmox Virtual Environment (PVE)</a>
        </li>
        <li>
            <a href="https://kubernetes.io/">Kubernetes</a>
        </li>
        <li>
            <a href="https://ceph.io/">Ceph</a>
        </li>
        <li>
            <a href="https://rook.io/">Rook</a>
        </li>
        <li>
            <a href="https://goharbor.io/">Harbor</a>
        </li>
        <li>
            <a href="https://cloudnative-pg.io/">CloudNativePG</a>
        </li>
        <li>
            <a href="https://verdaccio.org/">Verdaccio</a>
        </li>
        <li>
            <a href="https://prometheus.io/">Prometheus</a>
        </li>
        <li>
            <a href="grafana.com/">Grafana</a>
        </li>
        <li>
            <a href="https://matomo.org/">Matomo</a>
        </li>
        <li>
            <a href="https://github.com/kubernetes/ingress-nginx">Nginx Ingress controller</a>
        </li>
        <li>
            <a href="https://metallb.universe.tf/">MetalLB</a>
        </li>
        <li>
            <a href="https://www.isc.org/bind/">Bind</a>
        </li>
        <li>
            <a href="https://pi-hole.net/">Pi-hole</a>
        </li>
        <li>
            <a href="https://nginxproxymanager.com/">Nginx Proxy Manager (NPM)</a>
        </li>
        <li>
            <a href="https://uptime.kuma.pet/">Uptime Kuma</a>
        </li>
        <li>
            <a href="https://www.noip.com/">No-IP</a>
        </li>
        <li>
            <a href="https://www.godaddy.com/">GoDaddy</a>
        </li>
        <li>
            <a href="https://mailcow.email/">Mailcow</a>
        </li>
        <li>
            <a href="https://listmonk.app/">Listmonk</a>
        </li>
        <li>
            <a href="https://mixpost.app/">Mixpost</a>
        </li>
        <li>
            <a href="https://www.freepbx.org/">FreePBX</a>
        </li>
        <li>
            <a href="gethomepage.dev/">Homepage</a>
        </li>
    </ul>
    <div>
        <a href="#table-of-contents">Go to Table of Contents</a>
    </div>
</section>

<%- include('includes/footer.ejs') %>