Offsite backups 101

I’ve learned some hard lessons in the couple decades of tinkering with computers. One of the biggest is data loss. Like everyone else, I didn’t worry as much as I should have and it cost me.

Quick background: I had a RAID 5 setup with three drives (4TB NAS edition) and had just completed about 2 weeks of burn in without incident, total of about 5TB of data or so. I had just removed my backups to begin another data migration of the system that temporarily stored all these when suddenly the server failed.

Now, when I say it failed, it crashed (apparent BSOD) and immediately rebooted. Big deal, it’s windows, it happens. But then the raid drive didn’t show. Another reboot, no luck. Go into partition manager, and the RAID array was completely zeroed out and empty. This contained backups since 2006 to as recently as a couple months ago, including family photos, old college projects and school work. Some of this was redundantly backed up on MediaFire but not nearly all of it.

Fast forward to now, and I’ve gone paranoid. I now have the old NAS server with twin RAID 1 set ups to ensure backups on these, along with Resilio sync to further push this out.

The plan: Offsite backups at two remote locations using off the shelf desktops.

This was accomplished using Ubuntu Desktop on two identical systems, both using Resilio sync which is stored on a RAID drive at my place as well. All changes are Read-only on the remote systems to ensure nothing can write back to cause issues here as well. Additional to this, are multiple archives stored on other local systems as well. Paranoid? A little bit. Worth it? Always. Knock on wood I’ll never need them, but now they’re there if anything happens.

Arduino Wireless Logging, Post Four: The Graphs

When I got all this data logged and running, I wanted an easier way to show data than spreadsheets or queries. The end goal was actual graphs to show trends and what not. Introducing “phpgraphlib”. This is an open source piece of code built to implement charts/graphs into webpages with ease. This took a bit of work, but here’s the idea behind it:

  • Be able to load a web page and show trends/data
  • Pull this information on demand from a database

Once the tinkering was over, I had a working dashboard to pull two images and show them in a webpage that refreshes every 60 seconds. I ended up going a little overboard in customization, but it happens. Here’s the final build:

  • Built a script that runs two sets of  data and groups by hour: humidity and temperature
  • Have a cron running every 5 minutes to compile this data into two separate PNG images to pull up online
  • Have a dashboard that refreshes every minute and shows both of these images

At this point in time, you can see the graphs here:

Continue reading Arduino Wireless Logging, Post Four: The Graphs

Arduino Wireless Logging, Post Three: Mysql, PHP, and a little networking

Here’s where things started to get a little more complex. At this point, I have a wireless system to transmit data from multiple nodes to a central hub (node0) which collects and shows the data via serial connection.

The general idea/goal is this:

  • Arduino Nodes send data to central node
  • Central node is connected to home network and sends this data to a LAMP server (Linux-Apache-Mysql-PHP)
  • The server takes this data, breaks it into a mysql statement, and pushes it into the database to log

This was a pretty straightforward process but took a lot of planning before taking this on to ensure everything went smoothly. I’m not going into detail for setting up a LAMP server but there is a massive amount of information online for this. If you have an old computer kicking around you could run as a headless server or want to run it as a VM, it will take anywhere from 15-20 minutes to fully install and be ready for this.

I’ll break this down into different parts:

  • Mysql database set up (database/tables)
  • PHP script to accept the data and log this
  • PHP script to show information in webpage
  • Actual Arduino code to compile then send this

Continue reading Arduino Wireless Logging, Post Three: Mysql, PHP, and a little networking

Arduino Wireless Logging, Post Two: The Design

This is the second part of the series for my Arduino wireless project I’ve been working on for a couple weeks now. You can start reading from the first post:

We have the idea, now we have to build it. I have three Arduinos set up at this point in this manner:

  • (1) Set up as wireless receiver/hub; this will read data, print to Serial and log to ethernet eventually
  • (2) Transmitters which simply read data every ~5 seconds to blindly send to receiver (node 0)

This layout will allow me to test from a couple devices to make sure the layout of the node IDs will work properly and be read properly as well. I ran into my first set of issues at this point: I can only read one Arduino at a time off Serial from the IDE, so went and imaged an older Dell D630 to handle the receiving end of things to troubleshoot the payload issues I ran into.  Continue reading Arduino Wireless Logging, Post Two: The Design

Arduino Wireless Logging, Post One: The Idea

I’ve had a number Arduino systems for some time now along with a large number of RF24 wireless modules and DHT11 humidity/temperature sensors. The idea I’ve had for some time is this:

  1. Arduino with a DHT11 sensor and RF24 wireless device pull then transmit data.
  2. Data goes to secondary hub/master Arduino using an RF24 module along with an Ethernet adapter (Wiznet 5100)
  3. This master hub then pushes the data to a web server containing a mysql database to log and store the information

Simple right? Wrong. This has been a project for the better part of 8 months now with little working results. Right up until the amazing “maniacbug” user built the RF24-master library for Arduino to handle most of the payload and data transactions. At this time, I have a working prototype and will keep posting information on this project soon! Stay tuned!

Latest Addiction: FALLOUT SHELTER!!!!

So I go in waves of games, projects, ideas, etc. Latest is Fallout Shelter on Android and it is massively addicting. The basic concept is this: You are the Overseer of a Vault, and you need to build it to keep people happy and maintain resources while unlocking new costumes and weapons.

TL;DR: Sims with the wasteland housing.

So this game is a riot, and it was wonderful right up until it crashed and wiped away all my content, including paid for bits. Reached out for a refund but didn’t hear anything as of today (3 days later). Onto the next bit: Lucky Patcher is an ungodly useful tool in these situations. That’s all I’m saying 😉

Arduino Home Monitoring

Lately I’ve been engrossed in a project to add some monitoring to my home. This includes temperature and humidity readings, along with monitoring for windows and doors eventually. This will be built almost entirely using Arduinos wherever possible.

The Goal:
To have a remotely viewable home dashboard to monitoring the house

The Plan:
Use Arduinos and cheap parts from Amazon to build this and code myself

At this point, I’ve built some of the systems but greatly need to expand upon this. More updates to (hopefully) come!