2018 Edition: Running a Free Web Site Periodic Operation Script

Jun 15, 2018, 11:40 PM

about10mins to read

Table of Contents

Top
Requirements
Technology Selection
Bottom-Up Selection
Free Infrastructure Candidates
AppEngine Standard Environment
AWS Lambda
Heroku
Pythagorean Switch
Language
Infrastructure Environment Construction and Coding
Want to Test Locally
Want to Open in a New Tab
Heroku Setup
Rubocop
Summary

The eternal need to "periodically poke and poke a web site" has been around since ancient times. I have a memory of creating a tool on Heroku in the past, but recently, the need has resurfaced like a geyser.

It's been a while since I created the tool, and I'm sure more convenient services have emerged. So, I've been looking for a way to "run a script that periodically pokes a web site as cheaply as possible" in 2018.

Requirements

First, let's organize the requirements.

Log in to the web site once a day and perform simple operations like clicking
The operation will take no longer than 5 minutes
Credentials can be set at deployment, so there's no need to manage them on the script side
It should not be executable by others
Running costs should be as low as possible

Very simple, isn't it? If there's a simple and free method, that's ideal.

Technology Selection

Bottom-Up Selection

It's reasonable to consider the infrastructure first, given the following reasons:

The program is extremely small
- → It's okay to decide on a language based on the infrastructure
Selenium can be easily used in many languages
- → Same as above
It only runs for 5 minutes a day, so it doesn't need to be constantly running
- → We can narrow it down to whether it's cheap for constant execution or periodic execution

In 2018, there are many infrastructures that meet these requirements. Let's consider a few free services.

Free Infrastructure Candidates

AppEngine Standard Environment

Cron can be set up easily, and deployment is also simple. Moreover, background execution is free for up to 9 hours/month, so it seems like it won't cost much.

AWS Lambda

CloudWatch can be used to achieve periodic execution like cron. Moreover, up to 1 million invocations/month are free. However, the execution time limit is exactly 5 minutes, which is a bit suspicious.

Heroku

This old-timer is still very practical (amazing). At least 550-1,000 hours/month are free, and the cron add-on is also free.

Pythagorean Switch

This refers to a dark technique of combining various small processes to achieve the goal, like using IFTTT to trigger a curl command when the door opens at home... It's fun to do as a hobby, but it's not maintainable, so I'll pass on it this time.

There are several options, but Heroku seems to be the most suitable for our purpose, and it has a sufficient free tier. In the end, I chose Heroku again.

Language

Heroku supports various languages, but I decided on Ruby, considering Heroku's support, Selenium's handling, and ease of writing.

Infrastructure Environment Construction and Coding

I gave it a grand title, but it's quick with Heroku. Additionally, some extra settings are required to use Headless Chrome on Heroku, but it's easy to follow the instructions in the following article:

It's a bit lonely, so I'll write a few coding tips.

Want to Test Locally

While the production environment is Headless, I want to check the actual operation during development. This can be easily achieved by setting an environment variable for testing and using a normal Chrome browser if it's set, or Headless if not.

def selenium_driver
  if ENV.key?('TEST')
    Selenium::WebDriver.for :chrome
  else
    caps = Selenium::WebDriver::Remote::Capabilities.chrome('chromeOptions' => { binary: '/app/.apt/usr/bin/google-chrome', args: ['--headless'] })
    Selenium::WebDriver.for :chrome, desired_capabilities: caps
  end
end

Want to Open in a New Tab

This is a function not provided by Selenium, so we need to hack it. We can achieve this by referring to the following StackOverflow post:

Heroku Setup

Setting up a Git repository for this Heroku environment is simple with the following command sequence:

cd <repository-path>

heroku login
heroku create --buildpack https://github.com/heroku/heroku-buildpack-ruby.git
heroku buildpacks:add https://github.com/heroku/heroku-buildpack-chromedriver.git
heroku buildpacks:add https://github.com/heroku/heroku-buildpack-google-chrome.git

heroku config:set SECRET=<XXXX>

heroku addons:create scheduler:standard

heroku git:remote -a <name-of-project>
git push heroku master

heroku addons:open scheduler

Rubocop

Let's do it.

Summary

I was surprised that Heroku still meets our needs. Even if we could do the same thing on GCP or AWS, I'd just think, "Wow, big companies are amazing," but it's unsettling that it's still free for so long.

I don't have any particular insights, but I hope this will be of some reference.