Bubble Docs
  • Introduction
  • New? Start Here
  • What is Bubble?
  • The Glossary
  • User manual
    • Getting started
      • What is Bubble?
      • Building your first app
        • Planning features
        • Database structure
        • Design and UX
        • eCommerce and payments
          • Shopping cart
          • Checkout page
          • One-time payments
          • Subscriptions
          • Marketplace
      • Creating and managing apps
      • The Bubble editor
        • Tabs and sections
          • Design tab
            • The element tree
            • The property editor
          • Workflow tab
          • Data tab
          • Styles tab
          • Plugins tab
          • Settings tab
            • Application settings
              • Custom headers/body
              • Visual settings
              • Social media sharing
              • Translating your app
              • Email settings
              • Collaboration
            • Custom domain and DNS
          • Logs tab
        • Tools
          • Key features
          • The search tool
          • The Issue Checker
          • The element tree
          • The element property editor
          • The debugger
          • Notes
        • Previewing your app
      • Transitioning to Bubble from...
        • JavaScript
        • HTML and CSS
        • SQL
    • Design
      • Elements
        • The element hierarchy
          • The element tree
        • The page
        • Containers
          • Groups
          • Repeating groups
          • Table elements
          • Popups
          • Floating groups
          • Group focus
        • Visual elements
        • Input forms
          • Text and numbers
          • Dates and time
          • File uploads
          • Selection controls
        • Reusable Elements
      • Styling
        • Color variables
        • Font variables
        • Styles
        • Custom Fonts
      • Responsive design
        • Building responsive pages
        • Legacy articles
          • The Basics (Legacy)
          • Building Responsive Pages (Legacy)
          • Migrating Legacy Pages
          • Tips When Designing (Legacy)
      • Templates
      • The Component Library
      • Importing from Figma
    • Data
      • The database
        • Data types and fields
        • Creating, saving and deleting data
        • Finding data
        • Displaying data
        • Protecting data with privacy rules
        • The database editor
        • Export/import data
          • Exporting data
          • Importing data (CSV)
        • Working with location data
        • Using Algolia
        • Database structure by app type
          • Marketplace Apps
          • Directory & Listings Apps
          • Social Network Apps
          • SaaS Apps
          • Project Management Apps
          • CRM Apps
          • Professional Services Apps
          • On-demand Apps
          • Documentation/ CMS Apps
          • Applicant Tracking System (ATS) Apps
          • Portfolio Apps
          • Gallery Apps
          • Online Store / Ecommerce Apps
          • Blog Apps
          • Messaging App
          • Dashboards
          • Building Block Apps
          • Bubble as a backend
      • Files
      • Images
      • Static data
        • App texts (translations)
        • Option sets
      • Temporary data
        • Custom states
        • URL parameters
      • User accounts
        • Authentication plugins
          • Facebook plugin
          • Fitbit plugin
          • Google plugin
          • Instagram plugin
          • LinkedIn plugin
          • Pinterest plugin
          • Slack plugin
          • Wistia plugin
          • YouTube plugin
        • Cookies set by Bubble
      • Time, dates and time zones
    • Logic
      • The frontend and backend
      • Workflows
        • Events
          • Frontend events
            • Recurring workflows
            • Custom events
          • Backend events
            • Database trigger events
        • Actions
        • API Workflows
      • Dynamic expressions
      • Conditions
      • Navigation
        • Single-page applications (SPA)
        • Multi-page applications
        • Page slugs
    • Workload
      • Understanding workload
        • Activity types
        • The workload calculation
        • Client-side and server-side processing
      • Tracking workload
        • Measuring
          • Using App Metrics
        • Monitoring
          • Workload notifications
          • Infinite recursion protection
      • Optimizing workload
        • Optimization framework
        • Optimization checklist
          • Page load
          • Searches
          • Workflows and actions
          • Backend workflows
        • Agency showcases
          • Minimum Studio
          • Neam
          • Support Dept
    • Security
      • Bubble's security features
      • Planning app security
      • Client-side and server-side
      • Bubble account security
      • App security
      • Page security
      • Database security
      • API security
        • API Connector security
        • Data API security
        • Workflow API security
      • Flusk
        • Overview
        • Flusk plan features
        • Getting started with Flusk
        • Flusk security tools
          • The Issues Explorer
          • Issue details
          • Tools and settings
            • Pages rating
            • Database rating
        • Flusk FAQ
      • Cookies
      • Security checklist
    • Publishing your app
      • Web app
      • Native mobile app
        • Global native mobile settings
        • iOS App Store
        • Google Play Store
        • Publishing FAQ
    • AI
      • Generate apps with AI
        • About AI app generation
      • AI page designer
      • Connect to AI agents
    • Maintenance
      • Collaborators
      • Version control
        • Best practices: Version control
        • Transitioning from the legacy version control
        • Terminology: Version control
        • Version Control (legacy)
      • Commenting
      • Database maintenance
        • Copying the database
        • Restoring database backups
        • Bulk operations
          • Bulk operation methods compared
        • Wiping change history
      • Performance
        • Hard limits
        • Capacity Usage (legacy)
        • Notes on queries
      • SEO
        • Introduction to SEO
        • SEO: App
        • SEO: Page
      • Testing and debugging
        • Introduction to testing and debugging
        • The debugger
        • The server logs
        • Supported browsers
      • API workflow scheduler
    • Integrations
      • API
        • Introduction to APIs
          • What is a RESTful API?
        • The Bubble API
          • Bubble API terminology
          • Authentication
            • How to authenticate
            • No authentication
            • As a User
            • As an admin
          • The Data API
            • Data API Privacy Rules
            • Data API endpoints
            • Data API requests
          • The Workflow API
            • Workflow API privacy rules
            • Workflow API endpoints
            • API workflows
              • Creating API workflows
              • Scheduling API workflows
              • Recursive API workflows
              • API Workflow Scheduler
              • Case: Stripe notifications
        • The API Connector
          • Authentication
          • API Connector security
          • API guides
            • OpenAI
              • Authentication
              • Calls
                • ChatGPT
                  • Chat
            • Google Translate
              • How to setup Google API keys
          • Streaming API
        • API security
        • Plugins that connect to APIs
        • API Glossary
      • Plugins
        • What Plugins Can Do
        • Installing and using Plugins
        • Authentication plugins
        • Special Plugins
      • SQL Database Connector
      • Bubble App Connector
      • WorkOS
        • WorkOS SSO
        • WorkOS API
    • Infrastructure
      • Sub-apps
      • Bubble release tiers
      • Hosting and scaling
        • How Bubble hosting works
        • Scaling with Bubble
        • CDN (Cloudflare)
        • Bubble app names
        • Domain and DNS
      • Compliance
        • GDPR
        • SOC 2 Type II
        • HIPAA
        • Other frameworks and standards
    • Bubble for Enterprise
      • Hosting and infrastructure
        • Dedicated instance
          • The Dedicated editor experience
          • Technical specs
          • Main cluster dependencies
          • Customizable options
          • Migration process
            • Pre-migration
            • During migration
            • Post-migration
      • Security and compliance
        • Single sign-on (SSO)
        • GDPR
        • SOC 2 Type II
        • HIPAA
        • Other frameworks
        • Bubble's security features
      • Admin and collaboration
      • Priority support
      • Billing and Payment Guideline for Dedicated Instances
  • Core Reference
    • Using the core reference
    • Bubble's Interface
      • Design tab
      • Design tab (Legacy)
      • Workflow tab
      • Data tab
      • Styles tab
      • Styles tab (Legacy)
      • Plugins tab
      • Settings tab
      • Logs tab
      • Template tab
      • Toolbar
      • Top and context menu options
      • Deployment and version control
        • Deployment & Version Control Dropdown (legacy)
      • Notes
    • Elements
      • General properties
      • General properties (Legacy)
      • Styling properties
      • Styling Properties (Legacy)
      • Responsive Properties
      • Responsive Properties (Legacy)
      • Conditional formatting
      • States
      • Page Element
        • Page Element (Legacy)
      • Visual Elements
      • Containers
      • Container Layout Types
      • Containers (Legacy)
      • Input Forms
      • Reusable Elements
      • Element Templates (legacy)
    • Workflows
    • Events
      • General events
      • Element events
      • Custom events
      • Recurring event
      • Database trigger event
    • Actions
      • Account
      • Navigation
      • Data (things)
      • Email
      • Element
      • Custom
    • Data
      • Data Sources
      • Operators and comparisons
      • Search
      • Privacy
    • Styles
    • API
      • The Bubble API
        • The Data API
          • Authentication
          • Data API endpoints
          • Data API requests
        • The Workflow API
      • The API Connector
        • Authentication
        • Adding calls
    • Bubble-made Plugins
      • AddtoAny Share Buttons
      • Airtable
      • API Connector
      • Blockspring
      • Box
      • Braintree
      • Bubble App Connector
      • Chart.js
      • Circle Music Player
      • Draggable Elements
      • Dropzone
      • Facebook
      • Fitbit
      • Full Calendar
      • Google
      • Google Analytics
      • Google Optimize
      • Google Places
      • Ionic Elements
      • iTunes
      • Slidebar Menu
      • LinkedIn
      • Localize Translation
      • Mixpanel
      • Mouse & Keyboard Interactions
      • Multiselect Dropdown
      • Progress Bar
      • Rich Text Editor
      • Rich Text Editor (Legacy)
      • Screenshotlayer
      • SelectPDF
      • Slack
      • Segment
      • Slick Slideshow
      • SQL Database Connector
      • Star Rating
      • Stripe
      • Tinder-like Element
      • Twitter
      • YouTube
      • Zapier
    • Application Settings
      • App plan
      • General
      • Domain / email
      • Languages
      • SEO / metatags
      • API
      • Collaboration
      • Sub-apps
      • Versions
  • Account & Marketplace
    • Account and billing
      • Pricing and plans
        • Plans and billing
        • Billing cycle
        • FAQ: Pricing and Workload
      • Account Management
      • Building Apps for Others
      • Selling on the Marketplace
      • Plans & Billing (legacy)
    • Official Bubble Certification
      • Hiring certified developers
    • Building Plugins
      • The Plugin Editor
      • General Settings
      • Updating to Plugin API v4
      • Adding API Connections
      • Building Elements
      • Building Actions
      • Loading Data
      • Publishing and versioning
      • Github Integration
    • Building Templates
    • Application and data ownership
    • Marketplace policies
    • Bug reports
  • Vulnerability Disclosure Policy
  • Beta features
    • About the Beta features section
    • Native mobile apps 🔒
      • Introduction
        • What is a native mobile app?
        • Native mobile vs. web development
        • Differences in native and web elements
        • Native mobile app terminology
      • Building
        • Views and navigation
        • Native mobile actions
        • Components and gestures
        • Device resources
          • Location services
          • Camera/photo library
      • Previewing
      • Publishing
Powered by GitBook
On this page
  • Introduction
  • What do we mean by bulk operations?
  • The two methods in short
  • Schedule API workflow on a list
  • Recursive workflows
  • Comparison table
  • Which method should I use?
  • Workload and performance
  • Parallel versus sequential processing
  • Setting a dynamic timestamp on each iteration
  • Reliability
  • The workflow scheduler

Was this helpful?

  1. User manual
  2. Maintenance
  3. Database maintenance
  4. Bulk operations

Bulk operation methods compared

This article compares the two methods for large bulk operations.

Last updated 11 months ago

Was this helpful?

In this article, we’ll explore the two most common methods for scheduling bulk database operations in your app. The two methods are somewhat similar, but from a technical perspective they behave differently, and have different use cases.

This article focuses on performing bulk operations in your app, as opposed to in the Bubble editor. If you want to perform bulk operations from the Bubble database editor, check the article below:

Article:

Introduction

Before we go into the details of each method, we’ll start by defining what we mean by a bulk operation in this context, and give a quick overview of what the two methods are.

Throughout this article, we'll sometimes refer to the Schedule API workflow on a list method as SAWOL.

What do we mean by bulk operations?

In the context of this article, a bulk operation refers to executing a workflow for each item in a list. For instance, if you have a list of 5,000 users and need to update a specific field in the user data type for all of them, you'd run the workflow 5,000 times, once for each user, to apply the necessary change. The purpose is to automate repetitive tasks across a large number of records.

Both methods will be scheduled and executed as .

The two methods in short

Schedule API workflow on a list

The Schedule API workflow on a list action is a single action that instructs Bubble to run a given API workflow once for each thing in a list. If you use this method to schedule an API workflow on 5,000 things, Bubble will schedule the workflow as 5,000 separate API workflows that run independently of each other. One workflow does not wait for another to finish. Bubble will work its way through the list as fast as it can. If you set a delay in-between each workflow, Bubble will still schedule all the workflows in one operation, but add the specified delay to the schedule time. For example, let’s assume the following scenario:

  • The current time is exactly 12:00:00 am

  • You use Schedule API workflow on a list of 5,000 users

  • The first API workflow should run immediately

  • You define a delay of 1 second

  • The actual workflow takes 0.2 seconds to finish

Bubble will do the following:

  • Schedule the first workflow at 12:00:00

  • Schedule the second workflow at 12:00:01

  • Schedule the third workflow at 12:00:02

  • And so forth

Regardless of how long the first workflow takes to complete, the second workflow will start at the assigned time. As such, the workflows are not looping, but scheduled for a definite time in the future.

Furthermore, in cases where your app is rate-limiting the number of workflows running at any given time (e.g. you have many backend workflows scheduled for the same time), workflows may not run until there are resources available, in which case it will start after its scheduled time.

So even with an interval, there is no guarantee that the workflows will run one at a time, or in order. As such, this method should be used only in cases where the workflows are independent. See the sections below for more details.

Recursive workflows

Using recursive workflows can potentially lead to infinite recursion, resulting in significant workload unit (WU) consumption. Starting on July 1st, 2024, Bubble will apply a default setting to terminate recursive workflow chains at 10 iterations for all new apps.

This means you need to either disable this feature or set a higher limit (recommended) if you plan to use recursion, or else any recursive workflow chain will be terminated after 10 iterations.

Recursive workflows behave a bit differently. Instead of scheduling a list of workflows, you are technically scheduling just one workflow on one thing, and this workflow contains an action that re-schedules the workflow.

Let’s assume the same scenario as above:

  • The current time is exactly 12:00:00 am

  • You use Schedule API workflow, and include a list parameter of 5,000 users

  • The first API workflow should run immediately

  • You define a delay of 1 second in the rescheduling action

  • The actual workflow takes 0.2 seconds to finish

  • The Schedule API workflow action takes 0.1 second to finish

Bubble will do the following:

  • Schedule the first workflow at 12:00:00

  • Schedule the next workflow 1 second after the first workflow has finished: 12:00:01:03

  • Schedule the second workflow 1 second after the second workflow has finished: 12:00:02:06

As you can see, the time each workflow takes to finish, and the time it takes to schedule a new iteration, is added to the total time of the bulk operation. In other words, recursive workflows are slower, but you can ensure that no workflow runs before the prior workflow is done.

The short difference between the two, then, is that Schedule API workflow on a list runs in parallel, while recursive workflows run sequentially.

Comparison table

SAWOL
Recursive workflow

Processing

Runs in parallel

Can run sequentially

Spends less workload

Spends more workload

Performance

Faster

Slower

Actions needed to run

One

Same number as items in the processed list

Reliability

Will always attempt to run all workflows

May stop if an error is hit

Do not happen

Can happen

Iteration timing

Static

Which method should I use?

Now that we know the basic difference between the two methods, let’s dig deeper into which method you should use.

Workload and performance

First, we’ll have a look at how the two methods differ from the perspective of:

  • : how much workload is needed to finish

  • Performance: how long the process takes to finish

An informal benchmark test using Schedule API Workflow on a List to execute 100K workflows compared to scheduling them recursively, gave the results below:

Task
SAWOL
Recursive workflow

Delete 100K things

20–25 min

6–7 hrs

Copy 100K things

60 min

10 hrs

Modify 100K things

75 min

12 hrs

WU for scheduling 100K workflows

~12,000 (~0.12 per workflow)

70,000 (0.70 per workflow)

Numbers can differ substantially, based on what your workflow does, but as the general trend shows, Schedule API Workflow on a list is a more performant and workload-friendly operation.

There are a few reasons why:

Performance

When using Schedule API workflow on a list, the completion time is generally shorter because Bubble is capable of running multiple workflows concurrently. This means if you have Workflow 1, 2, and 3, Bubble can process them all at the same time, aiming to complete the task as quickly as possible. This parallel execution helps in speeding up the processing.

Secondly, using the SAWOL action will schedule all the workflows in one operation, whereas a recursive workflow will schedule each workflow separately. In other words, Schedule API workflow on a list is one action, while a recursive workflow (using our earlier examples of a list of 5,000 things) is 5,000 actions. Scheduling an API workflow is a fast process, but noticeable when multiplied by a long list of things such as 5,000.

Workload

Recursive workflows consume more server resources in two main ways.

  • Reschedule action: Firstly, they need that extra action to reschedule themselves, using additional server capacity.

  • Conditions: Secondly, each cycle typically requires a conditional check to prevent infinite looping, further increasing server load. Each of these factors contributes cumulatively to the total workload on the server.

Improperly configured recursive workflows can unexpectedly use up your allocated workload units, possibly exhausting them in a single operation. This risk arises if the condition meant to halt the indefinite looping of the workflow is absent or incorrectly set up. Without a properly functioning stopping condition, the workflow may continue until it has consumed all available workload units

Parallel versus sequential processing

As we touched upon earlier, Schedule API workflow on a list runs the workflows in parallel when possible. That doesn’t mean that all 5,000 run simultaneously, but generally that Bubble will not make any attempts to stop them from overlapping. This in turn means that even if you specify an interval (such as 1 second) between the workflows, they might still overlap if the prior workflow takes some time to finish. On longer lists, this is more likely to occur. As a result, setting an interval can stop workflows from overlapping, but there’s no guarantee.

Recursive workflows, on the other hand, can be guaranteed to run in sequence, since the action that schedules the next cycle is often the last action in the recursive workflow. That means the workflow first runs through the action steps, and the next cycle will not start until it’s scheduled.

While this can take longer and spend more server resources, it can be useful in scenarios where you want to avoid any overlap. We can illustrate this with a few examples:

Example 1: Race conditions

A race condition is a term often used in software development to describe when two or more processes access shared data and try to change it simultaneously. For example, if two Bubble workflows try to make changes to the same thing at the same time, the metaphor implies that those two processes are “racing” against each other to make the change, which can affect the outcome. This can lead to unexpected behavior, where it’s difficult or even impossible to predict which process will finish first.

If you are using Schedule API workflow on a list, and the workflow involves making changes not only to each individual thing in the list, but also to one thing that’s shared across more than one workflow, you may not get the expected result, since the order of execution is not guaranteed.

To avoid this, you can use a recursive workflow to run the operation sequentially, or, if possible, make the change to the shared thing only when all the workflows in the list have finished.

Example 2: depending on data from prior step

Sometimes, you’ll need to send a dynamic parameter along with the next scheduled API call. In cases where this parameter relies on information from the current cycle, you’ll need to be sure that the data is generated before the next iteration.

Consequently, you can’t schedule the workflows in one operation, but will need to schedule them one by one recursively. While this can be necessary and useful in some scenarios, finding a workaround that doesn’t force you to send unique parameters for each iteration can have a big effect on the performance and workload consumption of the process.

Example 3: Tracking the progress

With Schedule API workflow on a list, it’s difficult to predict or confirm when a process is done, as there is no guarantee on order of execution and the workflows are all independent.

A recursive workflow gives you more flexibility to track the ongoing process, or to know when it’s finished. Typically the iteration number is passed as a parameter to a recursive workflow and decremented in each subsequent call to schedule the next one. This allows you to include actions in the workflow with conditionals that reference where the workflow is in the current sequence. This can be useful to show the progress visually on-screen, start a workflow when the process is done or communicate to the user (such as sending an email) when it’s finished.

Setting a dynamic timestamp on each iteration

Recursive workflows are not only used for bulk processing, but can also be used to perform a specific task at a given interval or time. For example, you may want to run a workflow at a dynamic time, such as the fifth day in every new month; by using a recursive workflow, you can have the workflow dynamically schedule itself to run again at a future time, potentially continuing this pattern indefinitely. This capability allows for flexible and precise timing in automating tasks.

Let’s say you’re using a workflow to schedule a regular email to users. By default, it's sent every month, but you may offer your users to change that schedule, such as setting a bi-monthly mail instead.

  1. If you use Schedule API workflow on a list, you will have to schedule each iteration on a set, unchanging schedule

  2. If the user changes their schedule setting, a recursive workflow can easily adapt to the new pattern

This is just one example to show that recursive scheduling can help you experiment more freely with dynamic patterns in execution. You could also have the time of the next iteration be based on data generated in the current iteration, keeping it truly dynamic.

Reliability

With Schedule API workflow on a list scheduling all workflows in one operation, they are guaranteed to run. A recursive workflow relies on the rescheduling action, and as such, can technically fail to move on to the next iteration. If one iteration stops, all upcoming iterations stop. This is not a bad thing in all scenarios – after all, if a workflow hits an error, you may want it to stop rather than to continue the work on all the things in the list. But if it's important that the workflow finishes, SAWOL will provide a slightly higher level of reliability.

When using SAWOL, all workflows are queued in a single operation, ensuring completion. In contrast, recursive workflows, depending on rescheduling actions, can occasionally fail to proceed to the next iteration due to various reasons. While this is rare, but not impossible, and there are a few reasons as to why it can happen:

  • excessive server resource consumption may cause timeouts, halting the rescheduling action

  • If one rescheduling action stops, it disrupts the entire cycle.

  • Server outages or errors can prevent the reschedule action from executing

The longer the list and timeframe of the full operation, the bigger statistical chance of an error is, making Schedule API workflow on a list a safer action. In cases of server errors, this workflow type simply pauses and then resumes once server resources become available again.

The workflow scheduler

The is Bubble’s built-in tool for keeping an eye on scheduled workflows. This tool allows you to see a list of scheduled workflow within a given timeframe, and take actions such as pausing or deleting them.

This process is more easily controlled with the Schedule API workflow on a list action, since it will simply list all the scheduled actions as soon as that action has completed. Recursive workflows, on the other hand, are “invisible” until scheduled. That is, Bubble doesn’t know that they will be scheduled until each iteration actually is.

This can make them more complicated to handle in the API workflow scheduler; by the time you have canceled one scheduled workflow, it may already have scheduled the next iteration, forcing you to update your search.

Article:

Infinite recursion protection
The bulk operations feature