Docs » Self Host Webhook

Beta software
Self-hosting Webhook is in beta and you use the software at your own risk. Specifically, we have not implemented release versioning and leave it up to you to pull in updates from master for bug fixes. This will change as the project matures.
A note about self-hosting Webhook

Webhook is a distributed platform built to manage thousands of websites. Our cloud-based architecture is a product of that requirement. Its intention was and always will be to make local development super simple for front-end developers who don't want to manage a lot of junk. For most, that's our hosted platform and its simple one-line npm install.

This self-host option is a different beast. It's for coders experienced with back-end architecture who want to dig deep into our guts. It's also our assurance to you that you will never need to worry about Webhook the company. The code will always be free and available to those that want to set up their own systems. Fork it, modify it and do what you will. But please don't complain that it doesn't install on a single linux box. It was never meant to.

What this guide covers

This document gives you step by step instructions to set up your own Webhook system. Depending upon your level of familiarity with Linux and Google's Cloud services the whole shebang will take you two to three hours. By the end of this document you will have a system that will allow you to host dozens of Webhook powered sites from the same architecture.

Architecture

Webhook's code is available under a MIT open source license. The following guide will give you instructions to self-host your own Webhook system, which itself is split over several git repositories available on github.

Set up outside dependencies

To host your own Webhook backend you'll need to set up the following services.

  1. An Elastic Search server (Free and hosted options).
  2. Mailgun account (Free tier will be fine).
  3. Embedly (Free tier will be fine).
  4. Firebase (Free tier will work).
  5. Google Cloud Platform (Cheap, pay as you go $.12 a GB in bandwidth).

Webhook uses Elastic search to provide search at the CMS level. Elastic search itself is open source and can be installed on any server following their instructions. However, if you'd like an easier solution, Webhook.com itself uses Found.no, which provides a hosted Elastic Search service for a monthly fee. The instructions below will be for their service, but can be easily applicable to an open source installation.

  1. Create a new cluster using the default settings. Name it whatever you want.
  2. Go to Access Control in your dashboard and copy/paste their sample file into your ACL file. The important bit is that it creates a user named readwrite with read/write access to the server.
  3. Write down the user/pass for this user. You'll need it later.

2. Mailgun

Webhook uses Mailgun to send invite emails to users who you invite to use your CMS.

  1. Sign up for a Mailgun account.
  2. Verify your domain with Mailgun.
  3. Write down the API key.

3. Embed.ly

Webhook uses Embedly to easily convert various embeds and URLs into a more readible format. Its used anytime you post a video into the WYSIWYG or use the embed widget.

  1. Sign up for an embed.ly account
  2. Write down the API key.

4. Firebase

Webhook uses Firebase to store and retrieve the data you save in the CMS.

  1. Sign up for Firebase.
  2. Create a new Firebase with a unique name.
  3. Click in to manage that Firebase, then click Security on the left and copy and paste the security rules here into the security field.
  4. Write down the API key. It can be found in the Secrets section on the left side.
  5. Go into simple login on the left and select Email and Passwords. Enable it.

5. Google Cloud Platform

Webhook uses the Google Cloud Platform to host and manage the rebuilds for your site.

  1. Create a new Google project.
  2. Write down the name and id of your project.
  3. Once created go to Api -> Credentials and select Create a new Client ID. In that menu select Service Account and create the ID.
  4. Once created it will prompt you to save a JSON file to your computer. Note the location. You'll need this file later.
  5. Next you'll need to verify the domain name (for the website you plan on using) with Google.
  6. After verifying your DNS you will need to add the service account you created earlier as an owner on the site. To do this go to the DNS verification page and say "Add an Owner", then add the email of the service account you created earlier. It'll be something like long_string@developer.gserviceaccount.com. If you don't know what it is go to the credentials area in your google project. You'll find it there.
  7. Go back to your project page and add your billing information in the Billing and Settings section under your project.
  8. You need to create two storage buckets: One to hold the sites themselves and one to hold backups. They can be named anything, but must be unique. We recommend calling them your-company-name-sites and the other your-company-name-backups. Write down the name of these buckets. You'll need them later.

Set up the Webhook Server

Webhook will work on any server that can run node. However, because Webhook uses Google Cloud Storage, using Google's servers may reduce latency in uploads and reduce bandwidth costs, but is not required. Because of that, we'll use Google's Compute Engine service as our example.

1. Set up a virtual instance

In your Google project, click compute engine and vm instances. Use the the following settings. Note that Webhook requires a decent amount of RAM and using lower level machines is not recommended.

  • Machine-Type: Standard.
  • Traffic Allow http:// and https:// traffic.
  • Image: Debian.
  • External IP. Select the create a new static IP option.

2. SSH into your VM

While your VM is setting up, install the Google Cloud Compute tools for your computer. During installation you should tell it to install the Python and PHP app engine tools as well.

  • Run gcloud auth login to authenticate your computer with google.
  • Run gcloud config set project [your_project_id]
  • Run gcloud compute instances list. It should show the virtual instance you created earlier.
  • Connect to your instance using gcloud compute ssh my-instance-name --zone us-central1-a. It'll set up a ssh password which you can save to your keychain. You should end up SSHed into your instance with root privileges.
  • Create a user called webhook by running sudo adduser webhook.

3. Setup the server

How to access root.
This part of the guide has you navigating a linux environment as a user and as root. Anything in the instructions with sudo in front should be run as root. You can switch to root at any time by typing exit or sudo su. You can switch back to your webhook user by typing su - webhook.

You'll be using apt-get alot here. It's a smart idea to run apt-get update first. Though we mention what steps you'll need sudo for below, it'll make it much easier to just switch yourself over to root immediately by typing sudo su before starting.

  • Install our server dependencies with sudo apt-get install build-essential unzip git beanstalkd memcached supervisor iptables.
  • Install Node. We recommend doing this from source per the instructions below.
sudo apt-get install python g++ wget libssl-dev
mkdir /tmp/nodejs && cd /tmp/nodejs
wget http://nodejs.org/dist/node-latest.tar.gz
tar xzvf node-latest.tar.gz && cd node-v*
./configure
make
sudo make install
  • Install Grunt and Reap through npm using sudo npm install -g grunt-cli reap
  • Switch to the webhook user we created earlier. su - webhook.
  • Clone the webhook-server repository into the webhook home directory. git clone https://github.com/webhook/webhook-server-open.git. You can also set up github properly on your server and use a fork.
  • Run npm install inside the webhook-server repository.
  • Edit the Gruntfile.js and edit the options as specified.
  • Add the Webhook server tasks to supervisor. To do this you can use the premade supervisor configuration file by running as root cp /home/webhook/webhook-server-open/webhook.conf /etc/supervisor/conf.d/
  • You will need to extract the private key from the JSON file google had you download earlier. To do this, copy the JSON file to your server into the webhook-server-open directory. Then as the user webhook run the command grunt extractKey --file=your_file_name.json.
  • Finally in order for backup's to work properly, you will need to set up a cron job to run at the interval that you want backups performed, the cron job should run grunt backupCron in the webhook-server-open directory as webhook. We have an example cron tab in the repository called backup-cron.example. You can edit your cronfile by running crontab -e.
  • Set up iptables below
iptables -A INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 3000 -j ACCEPT
iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3000
  • Install iptables-persistent. sudo apt-get install iptables-persistent. Click yes on all the prompts.
  • Create the directory /var/beanstalk. Run mkdir /var/beanstalk as root.
  • Run sudo supervisorctl reread sudo supervisorctl update sudo supervisorctl restart all. You'll know everything is running ok if you run sudo supervisorctl status and everything shows as running.

After all this your server should be set up and running, but there is one more thing to set up.

Set up image resizing with Google App Server

In order for image resizing to work you'll need to upload the webhook-images Google App Engine project to your project. This project is in charge of getting the resizable url from Google.

  1. Clone the webhook-images repo to your local computer git clone git@github.com:webhook/webhook-images.git.
  2. Edit the app.yaml file in that directory and change the first line to application: your-google-project-id. This is the project id you created earlier.
  3. Next if you haven't already, run gcloud auth login to load up the SDK.
  4. In your webhook-images directory run appcfg.py update .. This will push the application up to your Google project.

Setup local development

Now that the Webhook server is set up, all you need to do is set up the site you want to develop on Webhook.

Create a new site using the Webhook CLI.

In order to do this you can use the same 'wh' utility we provide for our hosted services. Simply install it to your development machine using npm install -g grunt-cli wh.

Unlike our hosted product when using the command line tools in a self-host situation you need to explicity pass your firebase name and server ip to your wh commands.

For example, let's create our first site.

Note
When performing this next step it's very important to name your website's domain EXACTLY as you want it to appear in your browser. That means you need to choose whether you want a naked or www domain. It will impact your DNS configuration later.
wh -f <firebase_name> -s <server_ip> create www.yoursite.com

Modify your local site settings for self-hosting

After creating the site you will need to modify your local site to get the CMS to work. Here you have two options.

Option 1: Use Webhook's hosted Javascript (recommended)

Your site by default will generate a pages/cms.html page that points to the Javascript hosted by webhook.com. This is totally fine and makes sure you get the latest CMS updates to your site. However, by default those files point to our own Firebase and other hosted systems, which you'll need to change. You'll need to change the values in .firebase.conf in your site local site directory withe following options.

{
  "secretKey"   : "secret_token_should_be_here",
  "siteName"   : "your_wh_create_sitename",
  "firebase"   : "your_firebase_name",
  "embedly"    : "your_embedly_api_key",
  "server"     : "http://your.google.vm.ip",
  "custom"     : true
}

Option 2: Compile your own Javascript files

If for some reason you need to fork our CMS or don't want to point to our files you can clone the webhook-cms repo, and modify some variables in 'config/environment.js'.

Replace the following values in the ENV array:

dbName:     "your_firebase_name",
uploadUrl:  "http://your.server.ip",
embedlyKey: "your_embedly_key",
selfHosted: true

When finished run grunt dist to compile your javascript files, then copy the files from the dist folder wherever you'd like to host the files (for example, the sites /static/ folder).

Using this option means it's up to you to keep up with changes to the CMS code.

Deploying your sites

When you're happy with your local website you can deploy it the same way you do the hosted version of webhook. Again, add your custom parameters for your firebase and server location.

wh -f <firebase_name> -s <server_ip> deploy

This will trigger a build on your Webhook server. The server should build a new cloud storage bucket with your site assets inside. Further edits to the CMS from the live site will rebuild the site.

How you know everything worked

If everything went according to plan you should notice the following.

  1. You have a new google storage bucket that uses the name from your wh create. Usually it'll be something like www.sitenmame.com.
  2. On your VM server you should have a build-folders folder in your webhook user's home directory. Inside it should have the sitename you used when you ran wh create.

Domain management

Lastly you'll need to point your domain to Google. Add the following CNAME to your DNS records. These records will need to match the site you created earlier when running wh create. In our example, that was a www address, so we'll point our CNAME there.

www       CNAME    c.storage.googleapis.com

Troubleshooting

Here's a quick checklist of common solutions for errors that pop up when self-hosting Webhook.

  • Make sure your DNS is verified and pointing to Google Cloud Storage.
  • Confirm that your Google Service Account email address is added as a owner to your verified domain.
  • Try restarting webhook_server on your Google Compute Engine instance. You can do this by running supervisorctl restart webhook_server.
  • Check your .firebase.conf file to make sure the values are correct.
  • Make sure you run your webhook command line with the extra -s -f settings when running in self-host mode.