How to combine DelayedJob with Shopify API Limits gem
As anyone who has done any serious web application development using the cloud (in my case Heroku) to host an App supporting Shopify shops, you need to be aware of the API call limits of both the shop itself and across all the shops the App is installed in. The 300 | 3000 rule - you can call one shop 300 times continuously or make 3000 calls to all shops continuously.
We used to have to write wrapper code to monitor and count API calls on top of writing complex program logic which was clunky to say the least. The ShopifyAPI development team kicked in by providing API call limits in the response headers, allowing an App to smartly self-monitor these limits during processing.
This was great except the development team forgot that the ShopifyAPI gem was not providing ActiveResource headers, making this small bit of progress tough to use for most people. Along came @christocracy who quickly hacked a small modification to ActiveResource and the ShopifyAPI gem providing simple and non-intrusive monitoring methods in support of API calls. In addition to now being able to know where the App stands with these limits, it is also important to run longer running API calls in background jobs (AKA delayed jobs) so as to not block the web App from servicing other calls. How to take advantage of this wicked combination of DelayedJob and the ShopifyAPI credit limits?
Assume a user selects ALL their (Orders|Products) in their Shop, and this amounts to a total of 603 items (a list of ID numbers), just over twice a shops typical API limit for continuous calls. To send these 603 (Orders|Products) to the App I setup the shop(s) with an injected link in the (Orders|Products) Action Drop Down menu. At the moment Shopify Admin has a bug limiting selection to 50 items, but let's assume we can send all 603 ID's as a GET parameter to our App.
Our App receives 603 ID's and needs to "touch" each (Order|Product) once (or twice, eg: read/update). For example by doing this simple operation:
That burns 2 API calls right there. So we are in some trouble since we can only do 300 API call continuously.
I decided to send ALL 603 id's to my method that will enqueue the request to process this list into the Delayed Jobs table I use. Delayed Jobs are run automatically by Worker threads that spin themselves in and out of existence, so as not to burn the App's credit card charges for worker fees to swiftly. I use Heroku and workers cost a nickel each per hour. The key to a Delayed Job is that you can control when it runs, what runs, and it does not affect your App since it runs in it's own thread.
When a job runs, the object that is instantiated can have code that checks the number of available API calls it can make. If there available API calls, we can interact with the Shop. If there are not enough API calls to complete a desired process, we have do take some steps to ensure we complete all the tasks that were specified.
We are processing a list of (Orders|Products) items so we need to keep track of the ones we have already processed and the ones we have not. This is easy by keeping track of the index of where we are in the iteration of the list of items. We can spawn a new Delayed Job, and since we know we have only partially processed our list of items, we instantiate the new job with the items of the list we have not processed yet.
Additionally when setting up a new job we can also specify when to run the job. If we tell it to run 601 seconds in the future, we should have a fresh slate of API calls available if the limit was reset. That is the essence of the code provided in the following gist. Of course, other actions could also be taking place that delay the availability of API calls, so the process should continue itself until it receives enough calls to complete. This means jobs that start and encounter a 503 response (no API calls can be made temporarily) should spawn a new job in the future, and terminate themselves properly.
I tested this code out and it works well. It processes as many API calls as possible before hitting the limit, at which point it spawns a new delayed job to start 10 minutes later. Watching the logs for activity, 10 minutes pass and the next delayed job was picked up by the worker thread assigned to run jobs. This job completed before running out of API calls and terminated gracefully, thus leaving the App in a good state, the shop in a good state and all was well that ended well.
Thanks to Chris for his excellent shopify_api_limits gem for making this possible.