2018 Edition: Running Periodic Website Operation Scripts for Free

The age-old need to "periodically click on a website" - I remember creating something on Heroku a long time ago, but recently this demand has sprung up again like cockroaches. They always come back when you least expect it.

It's been quite some time since I last made a tool, and surely more convenient services have emerged. So, I decided to investigate the current methods in 2018 for "running a script to periodically click on a website as cheaply as possible."

Identifying Requirements

First, let's organize the requirements:

• Log into a website once a day and perform simple operations like clicking
• The operation should finish within 5 minutes at most
• It's sufficient if credentials can be set during deployment
  • No need for user authentication to be managed by the script
• Others cannot execute it arbitrarily
• Keep running costs as low as possible

Very simple, isn't it? If there's an inexpensive and easy method, that would be optimal.

Technology Selection

Bottom-up Selection

For the following reasons, it's reasonable to start by considering the infrastructure:

• The program size is extremely small
  → We can decide on the language based on the infrastructure
• Selenium is easy to use in many languages
  → Same as above
• Since it only runs for 5 minutes a day, it doesn't need to be constantly running
  → We can focus on finding options that are either cheap for constant execution or support periodic execution for just 5 minutes

As of 2018, there are many infrastructures that meet these requirements. Let's consider a few services that seem to be free among them.

Free Infrastructure Candidates

AppEngine Standard Environment

It's easy to set up cron jobs and deploy. Also, background execution is free for up to 9 hours/month, so it seems it won't cost anything.

AWS Lambda

Periodic execution like cron is possible when combined with CloudWatch. Also, it's free for up to 1 million invocations per month. However, the execution time limit is exactly 5 minutes, which seems a bit risky.

Heroku

An old-timer, but still extremely practical (amazing). You can run it for free for at least 550-1,000 hours/month, and the cron add-on is also free.

Rube Goldberg Machine

This isn't a specific service, but rather a dark technique of combining various small processes to achieve the goal in a Rube Goldberg machine-like manner. For example, using the opening of a house door as a trigger to hit curl via IFTTT... and so on. It's fun to do as a hobby, but it seems like it would become unmaintainable after 3 days, so I'll pass on this for now.

There are several options, but Heroku seems to be the most mature and has sufficient free tier. Considering it's the most suitable for this purpose, I ended up choosing Heroku again.

Language

While Heroku supports several languages, I decided on Ruby, balancing Heroku's support, Selenium's handling, and ease of writing.

Infrastructure Setup and Coding

I've given it a grand heading, but with Heroku, it's done in an instant. Also, there are a few additional settings needed to use a headless browser (Headless Chrome) on Heroku, but it's all laid out in the article below, so just follow it to complete:

http://katsulog.tech/regularly-run-chrome-headless-selenium-with-heroku/

Since this alone would be too brief, I'll write a few coding tips.

Want to Confirm Operation in Local Environment

While production uses Headless mode, during development, I want to see it actually running. This is easy to achieve; just set it up to launch regular Chrome if a test environment variable is set, and Headless if not:

def selenium_driver
  if ENV.key?('TEST')
    Selenium::WebDriver.for :chrome
  else
    caps = Selenium::WebDriver::Remote::Capabilities.chrome('chromeOptions' => { binary: '/app/.apt/usr/bin/google-chrome', args: ['--headless'] })
    Selenium::WebDriver.for :chrome, desired_capabilities: caps
  end
end

Want to Open in a New Tab

Since this is a feature not provided by Selenium, we need to forcibly rewrite it. You can achieve this by referring to the StackOverflow post below:

https://stackoverflow.com/questions/6421988/webdriver-open-new-tab/9949850#9949850

Heroku Setup

Setting up the git repository for this Heroku environment can be easily done with the following command sequence:

cd <repository-path>

heroku login
heroku create --buildpack https://github.com/heroku/heroku-buildpack-ruby.git
heroku buildpacks:add https://github.com/heroku/heroku-buildpack-chromedriver.git
heroku buildpacks:add https://github.com/heroku/heroku-buildpack-google-chrome.git

heroku config:set SECRET=<XXXX>

heroku addons:create scheduler:standard

heroku git:remote -a <name-of-project>
git push heroku master

heroku addons:open scheduler

Rubocop

Let's apply it.

Conclusion

I was surprised that Heroku is still suitable for this purpose. Even if the same thing can be done with GCP or AWS, I would just think "As expected of big companies," but being free for so long like this makes me a bit uneasy.

There's not much insight, but I hope this can be of some reference.