RAILS SCHEDULER ADJUST FOR TIME ZONE PATCH

Now, around this time, the reconciler pulled a list of jobs from Redis and tried updating the list with any potential new schedules. Any job registered with a schedule at 1:00:01PM or after will get executed on their next run time.į atal exception caused by a race conditionĪs mentioned above - with each deployment, when a process shuts down, sidekiq-scheduler clears the schedule next, when a new process starts up, the static schedule is loaded back in Redis by sidekiq-scheduler and the dynamic jobs are loaded by an internal job. This was because between 12:59:49PM - 1:00:01PM (12 seconds), there was no schedule registered in Redis. At 1:00:01 PM, a new process comes online and registers the schedule in Redis.ĭuring this time, any scheduled Action run that was supposed to run on the hour mark (1:00 PM) was missed.

At 12:59:49 PM, one of the new processes experienced a fatal exception and crashed. Around ~12:59 PM, old processes are shutting down, and new processes are coming up. It reconciles static and dynamic jobs to ensure the right jobs are in Redis to be queued on time.Īround 12:56 PM on this day, a rolling deployment had gone out. Lastly, our setup of sidekiq-scheduler runs an additional Ruby Thread in the background every 5 seconds let's call it the reconciler. The internal job adds the Action runs (dynamic jobs) by clearing the entire schedule first and then re-adding static & dynamic jobs. In addition, we also have an internal job that ensures all dynamic jobs are also in Redis when a new Sidekiq process boots. Sidekiq-scheduler only adds the static schedule in Redis. Sidekiq-scheduler clears the schedule (static and dynamic) in Redis. When a new dynamic job is added or updated by our customer, we append the same in Redis as well, so it is available to be queued by sidekiq-scheduler.Īs part of the shutdown and start-up events, a few things happen: We store the configuration for dynamic jobs as standard cron syntax in our database. These are primarily internal jobs that run on schedule.ĭynamic Jobs: These jobs represent scheduled Action runs by our customers.

Static Jobs: These are a finite set of Sidekiq jobs defined via a YAML file. Our system has both static and dynamic jobs. Our customers define the schedule for their Action runs via Tines product UI. Sidekiq-scheduler makes it easy to run jobs on a schedule. Sidekiq is a popular and reliable gem for processing async jobs in a Ruby/Rails application. We use an open-source Ruby gem called sidekiq-scheduler that takes care of scheduling jobs which are later processed by Sidekiq. Deep diveīefore we begin, it might be helpful to gain a bit of context about this part of the system. After a thorough deep dive, we compiled a list of things that happened simultaneously for this kind of scenario to present itself. We followed up with comms to all the customers impacted by this issue.Įven though we shipped a patch, not having a good mental model of the contributing factors didn’t sit right with us.

RAILS SCHEDULER ADJUST FOR TIME ZONE PATCH

At this time, we knew enough about where the problem originated from and immediately shipped a patch to ensure this same situation didn’t happen again. Hence, the missed Action run that our customer correctly pointed out. We found a smoking gun from our logs that indicated a fatal exception, leading to a set of Action runs not getting executed on the next hour mark. We spun up an incident and got all hands on deck to investigate, gather evidence and mitigate the incident as soon as possible.Īt the time of the incident, the primary cause wasn’t entirely clear. So, we immediately started looking and soon realized this issue impacted more than one customer. On a calm Tuesday evening, one of our customer support engineers surfaced a customer flagged issue - our customer noticed one of their Actions that was supposed to run once a day hadn’t on the given day 😱. In this post, we examine how our old job scheduler system worked, a very interesting race condition, and why we replaced our old scheduler with something more reliable to meet our delivery guarantees. Our customers rely heavily on this feature in carrying out their mission-critical workflows. For example, an HTTP Request Action that runs every minute, once a day, or every few hours. One commonly used feature with Tines is the ability to configure your Actions to run on a schedule ( docs).

Rails scheduler adjust for time zone