Looking forward to dive into next week’s Cloud Musing newsletter on dynamic scheduling. This week, I will talk about User Acquisition and Growth. If you like this kind of content subscribe to our newsletter: https://lnkd.in/e3Xj4qhG
Dynamic scheduling is one of the biggest challenges developers face. For example, I want to deploy the Oatfin API on February 10, 2023 at 3:00 AM. Dynamic scheduling is hard because there is little support for it out of the box. With that also, as every developer knows, timezone is very hard to deal with.
Static scheduling on the other hand is very simple, every programming language provides some kind of support for writing cron jobs. An example of a cron job: I want to import data from a vendor every day at 8:00 PM.
Here is very high level on how we deal with this problem:
For the frontend:
1. To keep it simple, a user specifies year, month, day, hour, and minute from their own point of view.
2. We use the moment-timezone npm package to guess a user’s timezone from the browser.
Backend:
1. We run 3 docker containers: celery, scheduler and api. Celery is the base image and the other 2 containers extend the base image and overrides the CMD directive in the Dockerfile.
2. We use MongoDB to store the exact schedule in the database with the user’s timezone.
3. We use celery as a task queue and celery-redbeat as the scheduler. Celery-redbeat provides a RedBeatSchedulerEntry object to insert schedules in Redis. When we insert a schedule in Redis, we translate the user’s schedule to UTC date and time.
4. Once the task is complete, we mark it complete in MongoDB, which removes it from the list of scheduled deployments.
5. When a user cancels a task, we delete this entry from Redis and it won’t run.