Databases run into this problem whenever a query filters on lower(email), on a date pulled from a timestamp, or on something derived from two columns. If an index holds only the raw column data, the optimizer does not always have a direct way to search by that transformed result. An expression index fixes that by storing the result of the expression inside the index, which lets the search target the same transformed result named in the predicate. PostgreSQL calls these indexes indexes on expressions, Oracle calls them function-based indexes, MySQL supports functional index parts, SQL Server usually gets there through indexed computed columns, and SQLite supports indexes on expressions.

Subscribe now

What an Expression Index Really Is

Most indexes store values taken straight from table columns and arrange those values so the database can search them quickly. An expression index changes that arrangement by storing the result of an expression instead. That stored value could be a concatenated name, a numeric calculation, or some other derived result built from one or more columns. Because the index holds the computed result rather than only the source column data, the optimizer can search for the transformed value named in the query instead of recalculating that value row by row.

Names differ across database engines, but the idea stays very close. PostgreSQL refers to indexes on expressions. Oracle uses the term function-based index. MySQL supports functional key parts. SQL Server usually gets there through computed columns that can be indexed. SQLite allows expressions inside CREATE INDEX in place of plain column names. Syntax changes from engine to engine, yet the storage idea remains the same. The index is built from the value produced by the expression, so a plain index on first_name is not the same as an index on first_name || ' ' || last_name, and an index on amount is not the same as an index on abs(amount). Those indexes are ordered by different values, which means they support different searches.

Same Idea Across Database Engines

For major SQL products, expression indexing comes down to precomputing the value the query wants to search. PostgreSQL lets you place the expression directly inside the index definition. Oracle does the same through function-based indexes. MySQL also supports direct expression-based index parts, though its rules have their own wording and syntax details. SQL Server takes a nearby route by defining a computed column first and then indexing that computed column. SQLite allows expressions directly inside CREATE INDEX as well. Different spellings and DDL forms can make the feature look less related than it really is, yet the stored value is still a computed result rather than raw column data.

PostgreSQL makes that idea easy to see with a full-name lookup. Instead of indexing first_name and last_name separately, the index stores the concatenated result, which means the query can search the same value it asks for.

CREATE INDEX people_full_name_idx
ON people ((first_name || ' ' || last_name));

SELECT *
FROM people
WHERE (first_name || ' ' || last_name) = 'Alex Carter';

That index is ordered by the combined full-name value, not by first_name alone and not by last_name alone. If the query filters on that same concatenation, the optimizer has an indexed value that matches the predicate directly.

SQL Server reaches the same destination in a different form. Rather than placing the expression directly inside CREATE INDEX, you first add a computed column and then build the index on that column.

ALTER TABLE people
ADD full_name AS CONCAT(first_name, ' ', last_name) PERSISTED;

CREATE INDEX ix_people_full_name
ON people(full_name);

SELECT *
FROM people
WHERE full_name = 'Kaitlyn Brooks';

What changes here is the syntax, not the idea. The database is still indexing a derived value rather than the raw source columns. The computed column simply gives that derived value a named place in the table definition before the index is built.

MySQL also supports direct expression-based indexing, and its syntax has its own small rules. Parentheses are part of the definition, and regular column parts can appear in the same index beside functional parts. That puts MySQL closer to PostgreSQL and SQLite in how the statement looks, while the internal storage route is tied to hidden virtual generated columns. All of these engine-specific details matter at DDL time, but the larger point stays steady. The index stores the result of a calculation, string expression, or function call so the search can target that computed value directly.

The Query Must Match

Matching the query text to the indexed expression is where this topic becomes more exact than people usually expect. SQLite is very strict about this. Its planner looks for the same expression form in the WHERE clause or ORDER BY clause that appears in the index definition, apart from very small syntax differences such as whitespace. That means mathematically equivalent expressions are not always treated as the same thing for index matching.

This example makes that easier to see:

CREATE TABLE t2(x, y, z);
CREATE INDEX t2_xy_sum_idx ON t2(x + y);

SELECT *
FROM t2
WHERE y + x = 22;

SELECT *
FROM t2
WHERE x + y = 22;

The two predicates return the same rows, but they are not written the same way. With SQLite, the index definition stores x + y, so the second query lines up with the indexed expression while the first query does not. That rule gives you a practical habit to keep in mind while writing queries. When an index is built on an expression, keep the predicate in the same form as the indexed expression instead of rewriting it into a nearby equivalent form.

MySQL has a similar idea with functional key parts. If an index is built on SUBSTRING(col1, 1, 10), the predicate needs to ask for that same expression form to benefit from the index. Change the arguments or alter the expression, and the match can disappear. That is an important part of how expression indexes behave. The database is not promising fast access for every related transformation. It is storing a particular computed value, and the query has to ask for that same value in a compatible form.

PostgreSQL follows the same general logic. If the index is built on (first_name || ' ' || last_name), then a predicate on that same concatenation is the natural fit for the index. If the index is built on upper(last_name), then a query filtering on upper(last_name) lines up with the stored value. Small wording changes in documentation from one engine to the next can make this feel more abstract than it is. What helps most is remembering what the index actually contains. It contains the result of the indexed expression, so the search has to target that same result rather than a loosely related version of it.

Good Fits for Expression Indexes

Some queries do not search raw column values. They search transformed text, a calendar date taken from a timestamp, or a numeric result computed from other columns. That is where expression indexes tend to help most. The common thread is repetition. When the same derived value keeps appearing in filters, sorting, or lookup logic, storing that derived value in the index gives the optimizer a direct route to the value named in the predicate. That does not mean every expression belongs in an index, but a recurring query form can line up very well with this feature.

Lowercased Text Searches

Case-insensitive lookup is one of the most natural places for an expression index. Text may be stored exactly as entered, with uppercase and lowercase letters preserved, while the search condition folds both sides to a common form through lower() or upper(). If the only index is on the original text column, the database may have to apply that text function during filtering before it can decide which rows match. When the transformed value is stored in an index, the search can target that stored form directly.

PostgreSQL can express that directly:

CREATE INDEX users_email_lower_idx
ON users (lower(email));

SELECT user_id, email
FROM users
WHERE lower(email) = 'alex@example.com';

That index is ordered by the lowercased email value, so the predicate is searching the same value stored in the index. When a table is queried this way again and again, indexing the transformed text can be far more helpful than keeping only an index on the original email column. PostgreSQL also allows a UNIQUE expression index here, which means a system can block entries that differ only by case rather than treating them as separate values.

Oracle reaches the same goal with a function-based index:

CREATE INDEX emp_last_name_upper_idx
ON employees (UPPER(last_name));

SELECT employee_id, last_name
FROM employees
WHERE UPPER(last_name) = 'BROOKS';

Searches like this are a strong fit because the expression is stable, repeated, and tied closely to the filter itself. A database can still search text without this type of index, but the engine has a far better chance of picking an indexed route when the transformed value already exists in index form. That becomes more helpful as the table grows and the same predicate keeps appearing in login checks, customer lookups, or administrative searches.

Date Buckets

Reporting queries frequently slice timestamps down to a day, month, or year. That sounds minor, yet it changes what the predicate is asking for. An index on order_ts is ordered by full timestamp values, while a filter on CONVERT(date, order_ts) or a related date-only expression is asking for a derived calendar value. If the same date bucket keeps appearing in filters, grouping, or joins, an expression index or an indexed computed column gives that derived value a place in indexed storage instead of forcing repeated recalculation during filtering.

SQL Server handles this through a computed column that can be indexed:

ALTER TABLE Sales
ADD order_day AS CONVERT(date, order_ts) PERSISTED;

CREATE INDEX ix_sales_order_day
ON Sales(order_day);

SELECT order_id, order_ts
FROM Sales
WHERE order_day = CONVERT(date, '20260501', 112);

Two details help frame why this form is useful. First, SQL Server places rules on indexed computed columns. The expression has to be deterministic, and index-column expressions also have to be precise. Second, Microsoft recommends explicit date conversion with a deterministic style when date literals are involved. That avoids string-to-date ambiguity and keeps the computed value in a form the engine can treat consistently.

PostgreSQL, Oracle, and MySQL can support this broader idea as well, though the syntax changes from engine to engine. The reason date buckets fit so well is tied to the query itself. If the business question keeps landing on the date pulled from a timestamp instead of the full timestamp value, the indexed value that helps most is the extracted date rather than the raw column.

Derived Numeric Values

Arithmetic expressions are another strong match. Filters and sorts sometimes care less about a stored number than about a value derived from it, such as the magnitude of a balance change, a discounted total, or a formula built from several columns. Oracle allows function-based indexes on arithmetic expressions, and SQLite gives a practical example with abs(amt). In both cases, the same idea applies. The query is not asking for the base value in its original form. It is asking for a computed numeric result, so indexing that result can be a better fit than indexing only the source columns.

SQLite’s abs(amt) example captures this well:

CREATE INDEX acctchng_magnitude_idx
ON account_change(acct_no, abs(amt));

SELECT acct_no, amt
FROM account_change
WHERE acct_no = 17
ORDER BY abs(amt) DESC;

That index does two jobs at the same time. It groups rows by account number and then orders them by the absolute value of the amount change. If the query keeps asking for rows in that form, indexing the derived numeric result gives the planner much better alignment with the requested order. The point is not limited to abs(). Similar thinking applies to recurring formulas such as price * quantity, subtotal - discount, or a rating score built from stored columns. What makes these expressions a good fit is repetition. The more the query logic returns to the same derived number, the more sense it makes to store that number in index form.

Write Cost Matching Rules Engine Limits

Read speed is only half of the decision. Every expression index has a write-side cost because the database has to compute the expression when relevant rows are inserted or updated. PostgreSQL states that expression indexes are relatively expensive to maintain because the derived value must be computed for each row insertion and for each non-HOT update. Oracle makes a similar point and states that function-based indexes on columns that are modified frequently are expensive for the database to maintain. That is why the best fit is usually a recurring expression that helps enough on reads to justify the added write overhead.

Engine rules can narrow what is allowed. SQL Server requires indexed computed-column expressions to be deterministic, and index-column expressions must also be precise. That rules out a range of expressions tied to float or real values, cross-row calculations, or nondeterministic behavior. It also places SET option requirements on sessions that create, change, and query these indexes. SQLite has a similar restriction in spirit. Functions inside expression indexes must be deterministic, and subqueries are not allowed. MySQL ties functional index parts to hidden virtual generated columns, which brings generated-column restrictions along with them. That means no subqueries, parameters, variables, stored functions, or loadable functions inside those indexed expressions. MySQL also does not allow primary keys to include these functional parts.

Oracle has a few limits worth keeping in view as well. A function-based index cannot contain NULL, so NVL may be needed when null-producing expressions are part of the search logic. Oracle also requires any user-defined PL/SQL function referenced by the index expression to be declared DETERMINISTIC, and if the function’s semantics change later, dependent function-based indexes need a rebuild so they stop reflecting the prior function behavior. Those rules are not just side notes. They affect daily query tuning, storage choices, and update cost. Expression indexes pay off best when the expression is stable, the predicate repeats, and the engine can legally store that derived result in a dependable form.

Conclusion

Expression indexes let a database store the result of a repeated calculation inside the index instead of relying only on raw column values. That changes how the optimizer can search, because predicates on lowercased text, date-derived values, or numeric formulas can match an indexed computed result rather than forcing extra row-by-row evaluation. The tradeoff is added index maintenance during inserts and updates, which is why they make the most sense when the same derived expression appears frequently enough to justify the extra storage and write cost.

1. PostgreSQL Documentation on Indexes on Expressions

2. PostgreSQL CREATE INDEX Documentation

3. Oracle Database Documentation on Function-Based Indexes

4. Oracle Database CREATE INDEX Documentation

5. MySQL Documentation on CREATE INDEX and Functional Key Parts

6. SQL Server Documentation on Indexes on Computed Columns

7. SQLite Documentation on Indexes on Expressions

Share Alexander Obregon's Substack

The idea of a covering index makes more sense when you tie it to what the database is actually doing during a query. It is not a separate index type with its own CREATE INDEX command. It is just a way to describe a case where one index already contains every column a query needs, so the database can return the result from the index itself instead of going back to the base table for extra column values. In SQL Server, that can happen with a nonclustered index. In PostgreSQL, it lines up with index-only scans when the needed columns are stored in the index. In MySQL, the same idea applies when an InnoDB query can be answered from index data alone.

Subscribe now

What Covering Means

Coverage belongs to a query, not to an index name by itself. A given index becomes covering only when it already stores every column that a statement needs to filter rows, join rows, sort rows, group rows, or return values to the client. Change the statement, and the answer can change with it. Some queries can be fully satisfied from an index, while closely related queries against the same table still have to reach back to the base table for extra column values. That distinction removes a lot of confusion around the term. People sometimes treat covering index as though it names a separate structure, but the phrase is really pointing to a match between a query and the columns stored in an index entry. The database is not switching into a different family of indexes. It is simply finding that the chosen index already holds everything needed for that statement.

Query First

Coverage starts with the statement itself. Begin with the SELECT list, then move to the search condition, then any join columns, sort columns, or grouped columns. If any part of that statement asks for a column that is not stored in the index entry, the index can still help locate rows, but it does not fully cover the query.

Take a small table with customer account data. Say the table has customer_id, email, phone_number, and created_at. An index on customer_id is useful for locating a customer quickly, but that does not mean every query for that customer can be answered from the index alone.

CREATE INDEX idx_customer_accounts_customer_id
ON customer_accounts(customer_id);

SELECT email
FROM customer_accounts
WHERE customer_id = 1201;

That index helps the database find the matching row for customer_id = 1201, yet the result still asks for email. If email is not stored in the index entry, the engine has to use the index to find the row and then visit the base table to fetch the missing value.

Now compare that with a second version:

CREATE INDEX idx_customer_accounts_customer_id_email
ON customer_accounts(customer_id, email);

SELECT email
FROM customer_accounts
WHERE customer_id = 1201;

This time, both the search column and the returned column are stored in the index. That gives the optimizer the option to satisfy the statement from the index itself without reading the table row for email.

Small changes to the query can remove that benefit right away. Watch what happens when the SELECT list grows:

SELECT email, phone_number
FROM customer_accounts
WHERE customer_id = 1201;

If the index stores only customer_id and email, the database still needs the base table to fetch phone_number. Nothing about the table changed, and nothing about the first two columns changed. The difference is that the query now asks for one more value. That is why the query comes first when you judge coverage.

Filtering rules follow the same idea. Returned columns are only part of the story. Search conditions count too, because the engine still has to evaluate them. Say a support table has ticket_id, status, opened_at, and agent_id. An index that stores status and opened_at can cover one statement but fall short on a very similar one.

CREATE INDEX idx_support_tickets_status_opened_at
ON support_tickets(status, opened_at);

SELECT opened_at
FROM support_tickets
WHERE status = 'OPEN';

That query asks for opened_at and filters on status, so both needed columns are already present in the index entry. Now look at this version:

SELECT opened_at
FROM support_tickets
WHERE status = 'OPEN'
  AND agent_id = 14;

Adding the filter on agent_id changes the result. If agent_id is not stored in the index entry, the database may still use the index to narrow the search on status, but it cannot stay in the index for the whole statement. It has to read the table row to check agent_id.

Sorting and grouping can affect coverage too. A query that returns one column and orders by a different one is asking for both. Grouped queries ask the engine to read grouping columns and aggregate input values. If those referenced columns are missing from the index entry, the index is no longer enough by itself. Coverage, then, is tied to the full statement rather than just to the SELECT list.

Why Index-Only Access Helps

Most index-based lookups follow a two-step flow. First, the engine walks the index structure to locate matching entries. After that, it uses the row locator stored in the index entry to fetch the base table row if some needed column is missing. That second trip is the part a covering index can remove.

Think about what the database has to touch during execution. Index pages are usually smaller than full table rows, so more of them fit in memory and fewer bytes have to move around when the query stays inside the index. Once the engine starts jumping from index pages to table pages for every matching row, extra reads and extra page visits enter the plan. On a small result set, the gain can be modest. On a statement that matches a long run of rows, skipping those table lookups can change cost in a noticeable way.

Take this query against a shipment table:

SELECT shipment_status
FROM shipments
WHERE tracking_number = 'ZX-88491';

If the chosen index stores tracking_number and shipment_status, the database already has the values it needs in the index entry. It can search for the tracking number and return the status without fetching the full shipment row. That keeps the amount of data touched tighter than a plan that has to leave the index and visit the table.

Now compare that with a broader request:

SELECT *
FROM shipments
WHERE tracking_number = 'ZX-88491';

A statement like that asks for every column in the row. Unless the index happens to contain the entire row, the engine cannot stay inside the index. It has to fetch the table row after finding the match. That is part of the reason narrow SELECT lists help so much. They give the optimizer a better chance to satisfy the statement from index data alone.

Coverage can also help with repeated lookups where the same narrow query runs again and again. Think of account checks, order status screens, or service calls that ask for a very small set of fields. If the statement requests only a few values and those values already live in the index entry, the engine can avoid extra table reads every time that query runs.

Engines do not all handle index-only access in exactly the same way internally, but the broad idea stays the same. The more fully the index can answer the statement by itself, the less the engine has to reach back to the base table. That does not mean a covered query will always be chosen by the optimizer, and it does not mean every engine can avoid every table touch in every storage condition. What stays true is the main point behind the term. Coverage helps by reducing or removing the need for that second trip from index entry to table row.

Read it as a storage question tied to a statement, and if the index entry already carries the values the query needs, the engine has fewer places to visit. If even one needed value is absent, the table visit returns. That is the whole reason coverage can have such a strong effect in day-to-day SQL tuning.

Included Columns Within Composite Index Design

Stored payload values and search-column order decide how wide an index becomes, how much data it carries, and what the optimizer can do with it. That is where included columns and composite indexes meet. Some engines let you store extra return values at the leaf level without making those values part of index navigation, while others reach the same result by placing extra columns directly in the composite index itself. The tradeoff stays the same. Wider indexes can satisfy more statements from index data, but they also take more space and add maintenance during inserts, updates, and deletes.

SQL Server INCLUDE

Within SQL Server, nonclustered indexes have a direct way to carry extra columns through INCLUDE. Those included columns live only at the leaf level, so they are stored with the final index rows rather than guiding navigation through upper B-tree levels. That separation is the whole point of the feature. Search columns belong in the indexed column list. Return-only columns usually fit better in INCLUDE, because they stay out of the ordered search sequence and out of the size limits that apply to indexed search columns.

Viewed from the storage side, the split is fairly practical. Search columns help the engine narrow the scan and preserve index order. Included columns are payload. SQL Server can read them from the leaf page after it reaches the matching slice of the index. SQL Server also allows included columns to hold some data types that are not valid as indexed search columns, though older large-object restrictions still apply in a few cases. That makes INCLUDE useful when the query returns columns that you do not want participating in index order.

Take an order history table like this:

CREATE INDEX IX_order_history_customer_date
ON order_history (customer_id, order_date)
INCLUDE (order_total, payment_status);

That definition puts customer_id and order_date in the searchable portion of the index. order_total and payment_status are carried as leaf-level payload. With that layout in place, a statement like this can benefit:

SELECT order_date, order_total, payment_status
FROM order_history
WHERE customer_id = 8843
  AND order_date >= '2026-01-01';

With that index, the columns that narrow the row set are still at the front, while the returned values are carried without becoming part of index order. If order_total had been moved into the indexed column list even though the statement never filters or sorts on it, the index would become wider in ways that do not help navigation.

Clustered tables add a detail that changes how nonclustered indexes are read. On a table with a clustered index, SQL Server folds the clustered index columns into each nonunique nonclustered index as the row locator. That means some statements can read clustered columns from a nonclustered index even when those columns are not listed directly in the nonclustered definition.

A billing table makes that easier to see:

CREATE INDEX IX_billing_account_due_date
ON billing_entries (account_id, due_date)
INCLUDE (balance_due);

If the table is clustered on invoice_id, SQL Server can carry that clustered column set in the nonclustered row locator. A query that returns invoice_id, due_date, and balance_due may already have access to those values from the nonclustered index row, depending on the final plan and index properties. That is useful, but it is still worth keeping these indexes disciplined. Very wide nonclustered indexes take more space, fit fewer rows on a page, and add extra maintenance when data changes.

PostgreSQL INCLUDE

Inside PostgreSQL, INCLUDE serves a very similar purpose. Included values are payload columns stored in the index, but they are not part of the search columns that guide index scans. That gives you a practical split between columns that help find rows and columns that are merely returned to the client. For unique indexes, the uniqueness rule applies only to the indexed search columns, not to the payload columns named in INCLUDE.

That distinction helps keep the searchable portion of the index tighter. It also avoids dragging return-only columns into the uniqueness rule or into the ordered scan path. Read it as a way to keep the navigation portion lean while still carrying extra values in the leaf tuple.

Take a user session table like this:

CREATE UNIQUE INDEX ux_user_sessions_token
ON user_sessions (session_token)
INCLUDE (user_id, expires_at);

With that index, uniqueness applies to session_token. The columns user_id and expires_at are payload values carried with the leaf row. A query that looks up a session by token and returns those two extra values can benefit from that split without making them part of the unique indexed search sequence.

Older PostgreSQL habits help explain why this feature is useful. Before INCLUDE existed, people sometimes pushed return columns onto the end of the indexed column list to get a covering effect. That older style still functions, but it also turns those return columns into part of the indexed search definition. Compare these two forms:

CREATE INDEX idx_product_lookup_old_style
ON product_lookup (sku, product_name, retail_price);

CREATE INDEX idx_product_lookup_sku
ON product_lookup (sku)
INCLUDE (product_name, retail_price);

The newer form keeps sku as the searchable column while storing product_name and retail_price as payload. That usually makes the intent easier to read and keeps upper index levels smaller than the old habit of pushing every returned value into the indexed column list.

Width still needs attention here because large payload columns can bloat the index, and PostgreSQL has tuple-size limits for index entries. If the leaf tuple grows too large for the index method, inserts can fail. Payload columns also do not participate in navigation, so there is little value in packing wide data into INCLUDE unless queries really benefit from reading those values from the index. PostgreSQL also keeps some boundaries around the feature. Expressions cannot be placed in INCLUDE, and support is limited to B-tree, GiST, and SP-GiST.

Column Order in Composite Indexes

Order within a composite index changes how the engine can read it. The first indexed column is not interchangeable with the second or third. Databases read composite indexes from left to right, so the early columns in the definition do most of the narrowing. That is why composite indexes usually begin with columns that the query filters most directly, then continue with columns that support more filtering or requested sort order. Payload values belong later in the definition, or in INCLUDE when the engine offers that feature.

The left-to-right rule is easier to follow with this inventory example:

CREATE INDEX idx_inventory_location_sku_status
ON inventory_snapshots (location_id, sku, stock_status);

That index can support lookups on location_id, on location_id with sku, and on all three columns in that same left-to-right sequence. It is much less useful for a search on sku by itself, because the index is ordered first by location_id. Column order is doing real physical sorting inside the index, not just listing names.

Sorting is affected too. If a query filters on an early column and requests rows ordered by the next indexed column, the engine may be able to read the rows from the index in the requested order. Reverse those columns in the index, and that benefit can disappear. That does not mean every composite index should be built around ORDER BY, but it does mean the sequence should reflect the statement forms you care about most.

Trailing payload values deserve restraint relatively speaking. Columns at the front of a composite index should earn their place by helping the engine find or order rows. Return-only values belong later, or outside the indexed sequence when INCLUDE is available. Pushing payload values too far forward makes the navigation portion wider without giving the scan much back in return.

MySQL Without INCLUDE

In MySQL, ordinary secondary indexes do not have a general INCLUDE clause, so the layout decision is more direct. If a query needs extra columns from the index, those columns usually have to be part of the composite index definition itself. That puts column order right in the middle of the planning process, because the same column list is carrying both navigation columns and any extra values you want available from the index.

The leftmost-prefix rule is what keeps that planning grounded. With an index on (store_id, item_code, price), MySQL can use the index for lookups on store_id, on store_id with item_code, and on all three columns in that order. A lookup that starts with item_code alone does not get the same benefit. That is why extra return columns usually belong near the tail end of the composite index, after the columns that truly narrow the row set.

Take this sales table example:

CREATE INDEX idx_sales_region_day_total
ON sales_daily (region_id, sales_day, gross_total);

That order gives MySQL a search path that starts with region_id, then narrows further by sales_day. The trailing gross_total value can still be read from the same index entry if the query returns it. Reverse the definition and start with gross_total, and the index becomes far less useful for the searches most people would run against that table.

InnoDB adds one more storage detail that changes how secondary indexes are read. Every secondary index record already carries the table’s primary identifier columns. That means a secondary index can sometimes satisfy a query that returns the primary identifier, even when you did not list that column in the secondary index definition. The tradeoff is storage. Long primary identifiers make every secondary index wider, because those values are copied into each secondary index record.

Read this example with that rule in mind:

CREATE INDEX idx_catalog_category_sku
ON catalog_items (category_id, sku);

If catalog_items has a primary identifier such as item_id, InnoDB stores that identifier in each secondary index row. Queries that search by category_id and sku can use this index, and a statement that also returns item_id may still stay within the secondary index data. That hidden carriage of the primary identifier is useful, but it also explains why a compact primary identifier is a good habit in InnoDB. Every extra byte there gets repeated across all secondary indexes on the table.

Conclusion

At the engine level, covering comes down to where the needed column values already live. If an index holds the search columns and the returned columns, the database can stay in that index for more of the statement instead of hopping back to the base table row. Included columns in engines that support them let extra return values ride at the leaf level without changing search order, while composite index order still controls how the engine narrows rows and reads them in sequence. Put those parts in the right place, and the index stops being only a shortcut to the row and starts carrying the data the statement needs.

1. SQL Server Indexes with Included Columns

2. SQL Server Index Architecture and Design Guide

3. PostgreSQL Index-Only Scans and Covering Indexes

4. PostgreSQL CREATE INDEX

5. MySQL Multiple-Column Indexes

6. MySQL InnoDB Clustered and Secondary Indexes

Share Alexander Obregon's Substack

Stored procedures can run fast on one call and then slow down on the next, even if the T-SQL text stays exactly the same. SQL Server compiles an execution plan, keeps that plan in memory, and tries to reuse it so it does not have to compile the same statement again and again. Reuse helps save CPU and can keep request volume moving, yet it also opens the door to trouble when a cached plan was built for one parameter value and then reused for a very different one. That is the point where plan reuse, parameter sniffing, and uneven data distribution start affecting one another. Current SQL Server versions still follow those same basics, but newer engine features and Query Store options give you better ways to test bad cases and keep them from dragging down later executions.

Subscribe now

How SQL Server Reuses Plans

After the first compilation, SQL Server tries to match later executions to a saved plan instead of building a fresh one every time. That match depends on more than procedure text alone. The engine checks the cached entry against the surrounding execution context, which includes things like database identity, session settings, and plan attributes tied to cache lookup. When those line up, SQL Server can run the saved plan again and avoid extra compile cost.

Memory is the second part of that story. For reuse to happen, the saved plan still needs to be present when the next execution arrives, and that depends on cache pressure and invalidation events such as schema or statistics changes. Reuse therefore comes down to two related checks. SQL Server needs a compatible cached plan, and that plan still needs to remain in memory when the next execution arrives.

Compilation Reuse Cache Keys

Matching starts differently for stored procedures and for ad hoc batches. A stored procedure gives SQL Server a stable object definition, which makes reuse more predictable. Ad hoc text is stricter. Small text changes, different qualification of object names, or different session settings can split cache entries. That is why a procedure call usually reuses plans more reliably than a batch sent with literals embedded directly in the text.

SQL Server also separates cached plans by plan store type. Plans tied to persisted objects such as stored procedures, functions, and triggers live in a different cache store from ad hoc or prepared statements. When you inspect cache DMVs, that split helps explain why two cached entries can look related while still landing in separate categories. Procedure-heavy workloads and literal-heavy ad hoc traffic usually leave different fingerprints in cache for that reason.

To watch reuse happen, start with a small procedure and run it twice:

USE tempdb;
GO

CREATE OR ALTER PROC dbo.usp_ObjectCount
AS
BEGIN
    SET NOCOUNT ON;

    SELECT COUNT(*) AS object_count
    FROM sys.objects
    WHERE type = 'U';
END;
GO

EXEC dbo.usp_ObjectCount;
EXEC dbo.usp_ObjectCount;
GO

Run a cache query after those calls so you can inspect the saved entry and its reuse count:

SELECT cp.usecounts,
       cp.objtype,
       cp.cacheobjtype,
       st.text
FROM sys.dm_exec_cached_plans AS cp
CROSS APPLY sys.dm_exec_sql_text(cp.plan_handle) AS st
WHERE st.text LIKE N'%usp_ObjectCount%';
GO

usecounts tells you how frequently SQL Server has reused that cached plan after it entered cache. objtype and cacheobjtype help identify what kind of entry you are looking at. If the plan stayed in cache between executions, the reuse count should rise instead of giving you separate one-use entries. That distinction is helpful early on because it separates true plan reuse from repeated compilation.

Cache lookup is not driven by T-SQL text alone. Plan attributes attached to the compiled plan tell SQL Server what has to match before reuse can happen. sys.dm_exec_plan_attributes exposes those attributes, and the is_cache_key column marks which ones participate in cache lookup. dbid, objectid, and set_options are familiar examples. set_options causes confusion in plenty of environments because two sessions can run the same procedure text while different plan-cache-affecting SET options lead SQL Server to keep separate entries.

This query exposes part of that attribute data for the cached procedure plan:

SELECT st.text,
       pa.attribute,
       pa.value,
       pa.is_cache_key
FROM sys.dm_exec_cached_plans AS cp
CROSS APPLY sys.dm_exec_sql_text(cp.plan_handle) AS st
CROSS APPLY sys.dm_exec_plan_attributes(cp.plan_handle) AS pa
WHERE st.text LIKE N'%usp_ObjectCount%'
  AND pa.attribute IN (N'set_options', N'objectid', N'dbid');
GO

Data from that DMV makes cache matching easier to follow because it shows that reuse depends on more than the procedure name. SQL Server also separates the compiled plan from the execution context tied to a specific call. The compiled plan can be shared across executions, while runtime state for a given call lives in a separate structure that is reinitialized before the next execution. Reuse of a compiled plan does not carry row-by-row runtime state from a prior call into the next one.

Plan Lifetime Inside Memory

Cached plans do not stay in memory forever. SQL Server keeps them there while they still have value and while memory pressure allows it. When memory pressure rises, the engine evaluates cached plans and removes entries whose current cost has dropped low enough to make eviction worthwhile. Frequently reused plans tend to last longer because SQL Server refreshes their standing when they are referenced again, while one-use entries are easier eviction targets.

Ad hoc entries get special treatment in that process. SQL Server starts an ad hoc plan with a current cost of zero, so a batch compiled and never reused can be removed quickly when memory pressure shows up. That behavior helps explain why literal-heavy ad hoc traffic can fill cache with entries that provide very little reuse benefit. On instances where one-time ad hoc text is a major source of cache churn, OPTIMIZE_FOR_AD_HOC_WORKLOADS can reduce wasted plan cache memory by storing a compiled plan stub on the first execution and saving the full compiled plan only after a later reuse.

Plan lifetime is affected by more than memory pressure. SQL Server also invalidates plans after changes that make the saved version unsafe or stale. Schema changes such as ALTER TABLE, ALTER VIEW, ALTER PROCEDURE, index creation or removal, and statistics updates can all lead to recompilation. Modern SQL Server versions handle recompilation at the statement level, so a single statement can be recompiled without forcing the entire batch through a full new compile every time.

For lab testing, a database-scoped cache flush is much safer than wiping the whole instance cache. That lets you watch first-execution behavior for a single database without disturbing unrelated cached plans elsewhere on the server:

USE tempdb;
GO

ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE;
GO

EXEC dbo.usp_ObjectCount;
GO

SELECT cp.plan_handle,
       cp.usecounts,
       cp.cacheobjtype,
       st.text
FROM sys.dm_exec_cached_plans AS cp
CROSS APPLY sys.dm_exec_sql_text(cp.plan_handle) AS st
WHERE st.text LIKE N'%usp_ObjectCount%';
GO

Running that sequence gives SQL Server a fresh starting point for the database, then places the procedure plan back into cache on the next execution. That is useful when you want to confirm that a later call is truly a reused execution instead of a leftover cached plan from earlier testing. Wider cache flushes still have their place, but they force unrelated queries to compile again later, which can create a burst of extra compile activity across the server.

Query Store helps fill a gap that the plan cache does not cover well. The cache is a live memory structure, so entries can disappear long before someone opens a DMV window. Query Store keeps query text, plans, and runtime stats over time, which makes it much more useful for historical review. Plan cache tells you what SQL Server has in memory right now. Query Store tells you what happened across a longer span, which is why the two features complement each other so well.

Why One Parameter Can Hurt the Next Execution

Trouble starts when SQL Server compiles a statement for the parameter value it sees at compile time and then reuses that same plan for a later call with very different selectivity. Stored procedures, queries sent through sp_executesql, and prepared queries all participate in that behavior. During compilation or recompilation, SQL Server sniffs the current parameter values and passes them to the optimizer so row estimates can be based on those values instead of on a generic average. Uneven value frequency is what turns that behavior into a performance problem. Data on one CustomerID can fill almost the whole table while most other CustomerID values return only a handful of rows. Statistics, density data, and histograms are what the optimizer reads to estimate row counts, and those estimates feed operator choice, memory grants, and join strategy. If the saved plan was compiled for a value from one end of that distribution, the next call can inherit estimates that are badly out of line with the rows it actually touches.

What Parameter Sniffing Means

Parameter sniffing is SQL Server reading the current parameter value during compilation or recompilation and handing that value to the optimizer before the plan is built. That behavior is normal and, in plenty of cases, beneficial. Procedures that always receive similarly selective values can get a better plan from a sniffed parameter than from a generic estimate. Trouble appears when row counts swing sharply between calls and the cached plan from the first compile is still the plan later executions inherit.

Take a lookup procedure that filters on CustomerID. If CustomerID = 1 returns nearly the whole table, the optimizer can lean toward a scan. If CustomerID = 199500 returns a single row, that same predicate can lean toward an index seek with a very different cost profile. SQL Server statistics hold value distribution data in histograms and densities, and the optimizer uses that data to estimate the row count a predicate is likely to return. That is why skewed data can pull the optimizer toward very different plans for different parameter values.

Local variables change the story a bit. SQL Server cannot sniff the runtime value of a local variable at compile time, so estimates fall back to average-density logic or heuristics instead of a specific incoming value. That can reduce the influence of the current parameter value, but it can also replace a targeted estimate with a broader average that is still not a great fit for the rows the query actually reads.

This rewrite makes that difference more easy to see:

CREATE OR ALTER PROC dbo.GetSalesByCustomer
    @CustomerID int
AS
BEGIN
    SET NOCOUNT ON;

    SELECT SalesID, CustomerID, OrderDate, Amount
    FROM dbo.SalesSkew
    WHERE CustomerID = @CustomerID;
END;
GO

CREATE OR ALTER PROC dbo.GetSalesByCustomerLocal
    @CustomerID int
AS
BEGIN
    SET NOCOUNT ON;

    DECLARE @LocalCustomerID int = @CustomerID;

    SELECT SalesID, CustomerID, OrderDate, Amount
    FROM dbo.SalesSkew
    WHERE CustomerID = @LocalCustomerID;
END;
GO

The first procedure gives the optimizer the incoming parameter value during compilation. The second hides that value behind a local variable, so the estimate is based on average-density behavior instead of a sniffed parameter. Neither form is universally better. The value distribution in the table is what decides which choice looks good and which one turns costly.

Lab Schema You Can Build

Reproducing this issue in a lab is far less messy than trying to infer it from a busy production database. Small lab data with heavy skew, an index on the filter column, and a stored procedure with a single equality predicate is enough to make the difference visible. SQL Server 2022 ties Parameter Sensitive Plan optimization to compatibility level 160, and Query Store is turned on below so you can inspect plan history after the tests. Query Store is on by default for new databases in SQL Server 2022, but an explicit setting keeps the lab self-contained.

Start with this setup:

USE master;
GO

DROP DATABASE IF EXISTS PlanReuseLab;
GO

CREATE DATABASE PlanReuseLab;
GO

ALTER DATABASE PlanReuseLab SET COMPATIBILITY_LEVEL = 160;
ALTER DATABASE PlanReuseLab SET QUERY_STORE = ON;
GO

USE PlanReuseLab;
GO

CREATE TABLE dbo.SalesSkew
(
    SalesID     int IDENTITY(1,1) NOT NULL
        CONSTRAINT PK_SalesSkew PRIMARY KEY,
    CustomerID  int NOT NULL,
    OrderDate   date NOT NULL,
    Amount      money NOT NULL,
    filler      char(120) NOT NULL
        CONSTRAINT DF_SalesSkew_filler DEFAULT REPLICATE('X', 120)
);
GO

CREATE INDEX IX_SalesSkew_CustomerID
    ON dbo.SalesSkew(CustomerID);
GO

;WITH N AS
(
    SELECT TOP (200000)
           ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS n
    FROM sys.all_objects AS a
    CROSS JOIN sys.all_objects AS b
)
INSERT INTO dbo.SalesSkew (CustomerID, OrderDate, Amount)
SELECT CASE
           WHEN n <= 199000 THEN 1
           ELSE n
       END,
       DATEADD(day, n % 365, '2024-01-01'),
       10.00
FROM N;
GO

UPDATE STATISTICS dbo.SalesSkew WITH FULLSCAN;
GO

CREATE OR ALTER PROC dbo.GetSalesByCustomer
    @CustomerID int
AS
BEGIN
    SET NOCOUNT ON;

    SELECT SalesID,
           CustomerID,
           OrderDate,
           Amount
    FROM dbo.SalesSkew
    WHERE CustomerID = @CustomerID;
END;
GO

That data load packs almost the whole table into a single CustomerID and leaves the rest of the values sparse. The nonclustered index on CustomerID gives the optimizer a real choice between a seek-driven plan and a broader scan-driven plan. UPDATE STATISTICS with FULLSCAN removes first-pass sampling noise, which makes the skew much more visible in estimated row counts.

Run the procedure in two different orders so the first compile sees different values:

ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE;
GO

SET STATISTICS IO, TIME ON;
GO

EXEC dbo.GetSalesByCustomer @CustomerID = 1;
EXEC dbo.GetSalesByCustomer @CustomerID = 199500;
GO

ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE;
GO

EXEC dbo.GetSalesByCustomer @CustomerID = 199500;
EXEC dbo.GetSalesByCustomer @CustomerID = 1;
GO

Execution order is the whole point of that test. If the first compile happens for the highly common value, the second call can inherit a plan that was priced for a large row count. If the first compile happens for the very selective value, the second call can inherit a plan that was priced for a tiny row count. Reads, CPU time, and the actual execution plan are what you want to compare between the two runs. Database-scoped cache removal is documented and is a safer lab reset than flushing the entire instance.

Fixes You Can Test

Targeted fixes are usually the best place to start. OPTION (RECOMPILE) tells SQL Server to compile a fresh temporary plan for that statement, use the current parameter values for compilation, and discard that plan after execution. For a query with highly uneven selectivity, that can turn a bad reused plan into a better per-call plan at the price of extra compile CPU.

This version applies that choice at the statement level inside the procedure:

CREATE OR ALTER PROC dbo.GetSalesByCustomer
    @CustomerID int
AS
BEGIN
    SET NOCOUNT ON;

    SELECT SalesID,
           CustomerID,
           OrderDate,
           Amount
    FROM dbo.SalesSkew
    WHERE CustomerID = @CustomerID
    OPTION (RECOMPILE);
END;
GO

Compile cost rises with that hint, but the optimizer no longer has to reuse a plan built for a very different parameter value. Statement-level RECOMPILE is also narrower than tagging the whole procedure with WITH RECOMPILE, which keeps the fix focused on the statement that is actually sensitive to skew.

OPTIMIZE FOR UNKNOWN is a second test worth running. This hint tells the optimizer to compile against average-density information rather than the current runtime value. That can be helpful when neither the highly selective plan nor the highly nonselective plan is a good candidate for broad reuse. It does not turn parameter sniffing off, but it does bypass the current runtime value for plan selection.

Try it like this:

CREATE OR ALTER PROC dbo.GetSalesByCustomer
    @CustomerID int
AS
BEGIN
    SET NOCOUNT ON;

    SELECT SalesID,
           CustomerID,
           OrderDate,
           Amount
    FROM dbo.SalesSkew
    WHERE CustomerID = @CustomerID
    OPTION (OPTIMIZE FOR UNKNOWN);
END;
GO

Average-density estimates can be useful in the middle ground, but they are still just averages. If the table has very sharp skew, a generic estimate can still miss badly for both the hot value and the rare value. That is why this hint is worth testing rather than assuming. In some cases, picking a representative literal with OPTIMIZE FOR (@CustomerID = 1) or some other typical value can land closer to the traffic you actually see.

Cases where code cannot be edited have a different route. Query Store hints let you attach certain hints to a captured query without changing the procedure text, and supported hints include RECOMPILE and OPTIMIZE FOR UNKNOWN. Literal OPTIMIZE FOR (@var = value) is not supported as a Query Store hint, so that particular form still belongs in the T-SQL text itself. Query Store hints are available in SQL Server 2022 and later.

This sequence finds the query_id and then applies a Query Store hint to that captured value:

DECLARE @query_id bigint;

SELECT TOP (1)
       @query_id = q.query_id
FROM sys.query_store_query_text AS qt
JOIN sys.query_store_query AS q
    ON qt.query_text_id = q.query_text_id
WHERE qt.query_sql_text LIKE N'%FROM dbo.SalesSkew%'
  AND qt.query_sql_text LIKE N'%WHERE CustomerID = @CustomerID%';
GO

EXEC sys.sp_query_store_set_hints
    @query_id = @query_id,
    @query_hints = N'OPTION(RECOMPILE)';
GO

Broader controls exist too. ALTER DATABASE SCOPED CONFIGURATION SET PARAMETER_SNIFFING = OFF disables parameter sniffing at the database scope, and USE HINT('DISABLE_PARAMETER_SNIFFING') does the same at the query level. SQL Server also documents OPTIMIZE FOR UNKNOWN as a query-level way to compile from average-density estimates instead of the current runtime value. Those are much wider choices than statement-level RECOMPILE, so they usually belong later in testing after narrower fixes have had a fair look.

What Newer SQL Server Versions Do

SQL Server 2022 introduced Parameter Sensitive Plan optimization, usually shortened to PSP optimization. Under compatibility level 160, that feature can keep more than one cached plan for a parameterized query when a single plan is not a good fit for all incoming values. Microsoft documents that PSP is turned on by default at compatibility level 160, and the feature exists precisely for nonuniform data distributions where a single reused plan can drag badly for part of the traffic.

You can inspect the relevant database settings with a quick query:

SELECT name, compatibility_level
FROM sys.databases
WHERE name = DB_NAME();

SELECT name, value, value_for_secondary
FROM sys.database_scoped_configurations
WHERE name IN
(
    'PARAMETER_SENSITIVE_PLAN_OPTIMIZATION',
    'PARAMETER_SNIFFING'
);
GO

Multiple plan variants do not mean the old tuning choices vanish. PSP has limits on the number of variants that can be kept for a dispatcher, and Query Store is still very useful for reviewing plan history and applying Query Store hints when a query needs extra direction.

That relationship can also run in reverse. If parameter sniffing has been turned off by trace flag 4136, by the database-scoped PARAMETER_SNIFFING option, or by USE HINT('DISABLE_PARAMETER_SNIFFING'), PSP optimization is disabled for that query traffic too. Compatibility level 160 by itself does not guarantee PSP engagement if parameter sniffing has already been disabled higher up in the stack.

Conclusion

SQL Server plan behavior comes down to compilation, cache reuse, memory pressure, and row estimates tied to parameter values. When the first compiled plan matches later calls, reuse saves compile cost. When data distribution changes the row counts the optimizer expects, that same reused plan can slow later executions. Looking at cache entries, statistics, execution order, Query Store, and Parameter Sensitive Plan optimization gives you a practical way to see why performance changes from call to call and how the engine decides what happens next.

1. SQL Server Query Processing Architecture Guide

2. Parameter Sensitive Plan Optimization

3. Query Hints in Transact-SQL

4. ALTER DATABASE SCOPED CONFIGURATION

5. Query Store Hints

6. Statistics in SQL Server

7. sys.dm_exec_cached_plans

8. sys.dm_exec_plan_attributes

Share Alexander Obregon's Substack

Relational databases use foreign keys to keep related rows consistent. A child row can reference a parent row, and the database checks that the link stays valid. Cascade actions tell the database what to do with matching child rows when the parent row is deleted or when a referenced value is updated, so the relationship does not end up pointing at something that no longer exists. Across major database engines, the usual action set includes NO ACTION, RESTRICT, CASCADE, and SET NULL, while SET DEFAULT is supported in some products but not all, so behavior can change between PostgreSQL, MySQL, SQL Server, and Oracle Database.

Subscribe now

How Cascade Rules Fit

Relationships in a relational database depend on rules that keep parent and child rows connected in a valid way. The database checks those links during inserts, updates, and deletes so a child row does not end up pointing at a parent row that is gone. Cascade actions belong to that same group of rules. They do not create the link, they tell the database how to react after the link already exists and the parent side changes.

Parent Rows

Every FOREIGN KEY relationship has two sides. The parent table stores the referenced value, while the child table stores the column that points to it. The foreign key clause is written on the child table, not on the parent table. Parent columns must be backed by a PRIMARY KEY or another uniqueness rule so the database has a stable target for every child reference.

Publishing data gives a good starting point:

CREATE TABLE publishers (
    publisher_id INT PRIMARY KEY,
    publisher_name VARCHAR(100) NOT NULL
);

CREATE TABLE books (
    book_id INT PRIMARY KEY,
    publisher_id INT NOT NULL,
    title VARCHAR(200) NOT NULL,
    CONSTRAINT fk_books_publisher
        FOREIGN KEY (publisher_id)
        REFERENCES publishers (publisher_id)
);

That layout places the parent value in publishers.publisher_id and the child reference in books.publisher_id. books depends on publishers in a very specific way. Every row in books must point to a publisher row that already exists, unless the child column is nullable and the schema allows it to be empty. The direction of the relationship does not change just because the child table has more columns or ends up with more rows. What defines the parent side is the fact that other rows point to it.

Data inserts make the relationship easier to read:

INSERT INTO publishers (publisher_id, publisher_name)
VALUES (10, 'North Harbor Press');

INSERT INTO books (book_id, publisher_id, title)
VALUES (101, 10, 'The Long Query');

INSERT INTO books (book_id, publisher_id, title)
VALUES (102, 99, 'Broken Reference');

The first INSERT into books succeeds because publisher_id = 10 is already present in publishers. The second INSERT fails because publisher_id = 99 has no matching row in the parent table. That check happens before any cascade rule becomes relevant. A cascade action does not change how the database validates a child row during insert time. Its job begins later, at the moment the referenced parent row is deleted or the referenced value changes.

Direction is what keeps this topic readable. Parent rows are not defined by age, size, or business importance. They are defined by reference flow. The parent table holds the value being referenced. The child table holds the column that refers to that value. Once that part is settled, every later rule becomes easier to read because ON DELETE and ON UPDATE are always reactions to changes on the parent side.

That parent child structure also explains why foreign keys are attached to relationship meaning rather than to row order. A row in books is not just placed near a row in publishers. It is tied to a specific publisher identity through publisher_id. If the publisher row is removed, the database has to decide what to do with the book row. If the publisher ID changes, the database has to decide what happens to child rows that still point to the old value. Those later decisions are what cascade clauses control, but they only make sense after the parent row and child row relationship is fully defined.

Referential Actions

Action clauses answer a narrow question. What should happen to matching child rows when a parent row is deleted, or when the referenced parent value is updated. The common family includes NO ACTION, RESTRICT, CASCADE, SET NULL, and in some database products SET DEFAULT. Support is not identical across engines, so the clause names may look familiar while product behavior still differs in important ways.

A school scheduling schema gives a good way to read those options:

CREATE TABLE instructors (
    instructor_id INT PRIMARY KEY,
    instructor_name VARCHAR(100) NOT NULL
);

CREATE TABLE office_assignments (
    office_id INT PRIMARY KEY,
    instructor_id INT UNIQUE,
    building_code VARCHAR(20) NOT NULL,
    CONSTRAINT fk_office_instructor
        FOREIGN KEY (instructor_id)
        REFERENCES instructors (instructor_id)
        ON DELETE CASCADE
);

CREATE TABLE class_sections (
    section_id INT PRIMARY KEY,
    instructor_id INT NULL,
    room_code VARCHAR(20) NOT NULL,
    CONSTRAINT fk_section_instructor
        FOREIGN KEY (instructor_id)
        REFERENCES instructors (instructor_id)
        ON DELETE SET NULL
);

CREATE TABLE payroll_profiles (
    profile_id INT PRIMARY KEY,
    instructor_id INT NOT NULL,
    tax_region VARCHAR(20) NOT NULL,
    CONSTRAINT fk_payroll_instructor
        FOREIGN KEY (instructor_id)
        REFERENCES instructors (instructor_id)
        ON DELETE RESTRICT
);

ON DELETE CASCADE removes child rows when the parent row is deleted. In that schema, an office assignment has no separate purpose after its instructor row is gone, so deleting the office assignment at the same time fits the relationship. ON DELETE SET NULL keeps the child row but removes the reference. A class section can stay in the table with instructor_id set to NULL, which leaves room for a later reassignment. ON DELETE RESTRICT blocks the parent delete while matching child rows still exist. Payroll data is treated more carefully there, so the instructor row cannot be removed until the child row is handled first.

Delete behavior gets the most attention, but update behavior belongs to the same family. ON UPDATE CASCADE tells the database to copy the new parent value into matching child rows. That can be useful in schemas where the referenced value is allowed to change. Not every major engine handles this the same way, which is why foreign key DDL is not fully portable across products. PostgreSQL, MySQL, and SQL Server support ON UPDATE actions on foreign keys. Oracle Database does not support an ON UPDATE clause in foreign key definitions, so a schema that depends on update cascades cannot be moved there unchanged.

This compact example brings the update side into view:

CREATE TABLE membership_tiers (
    tier_code VARCHAR(20) PRIMARY KEY,
    tier_name VARCHAR(50) NOT NULL
);

CREATE TABLE members (
    member_id INT PRIMARY KEY,
    tier_code VARCHAR(20) NOT NULL,
    full_name VARCHAR(100) NOT NULL,
    CONSTRAINT fk_members_tier
        FOREIGN KEY (tier_code)
        REFERENCES membership_tiers (tier_code)
        ON UPDATE CASCADE
);

With that definition in place, changing a parent value such as tier_code from SILVER to STANDARD causes matching rows in members to move to the new code automatically. Without ON UPDATE CASCADE, that parent update would be blocked if child rows still referred to the old value.

Default behavior also deserves careful mention. If no explicit delete action is given, the database usually blocks the parent delete when matching child rows still exist. The names NO ACTION and RESTRICT sound close, but they are not treated exactly the same in every product. PostgreSQL separates them in cases that involve deferred constraint checking, while MySQL treats NO ACTION as the same as RESTRICT. That means the text of the clause cannot be read in isolation from the database product where it runs.

SET NULL and SET DEFAULT carry their own limits too. SET NULL only makes sense if the child column is allowed to contain NULL. If that column is declared NOT NULL, the database cannot replace the parent reference with NULL, so the foreign key definition would not fit the column rule. SET DEFAULT has an extra requirement. The default value placed in the child column still has to satisfy the foreign key relationship after the change. In products where SET DEFAULT is unsupported or only partially supported, that clause is not a safe portable choice.

A good way to read referential actions is to tie each one to the meaning of the child row. If the child row should disappear with the parent, CASCADE fits. If the child row should stay but lose its reference, SET NULL can fit. If the parent row should be protected from deletion while children still exist, RESTRICT or NO ACTION is closer to that rule. The foreign key clause is where that decision is written down, and the database carries it out at the time the parent change happens.

Cascades Across Larger Schemas

Foreign-key actions stay fairly readable while a schema has only two related tables. Things get more serious when a delete or update can move through three, four, or ten links in a row. At that stage, a single statement can touch far more rows than it may suggest, and engine-specific limits start to affect what the database can do and how far a change can travel.

Multi Level Chains

Deletes can travel through parent, child, and grandchild tables in sequence. Remove a row at the top, and the database can keep moving downward through later references as long as each relationship uses ON DELETE CASCADE. That behavior is helpful when lower tables exist only because the row above them exists. After the parent goes away, child rows no longer have a valid reason to stay behind.

Catalog data makes that easier to read. Remove a catalog row, and related product rows can disappear. Remove those product rows, and related inventory event rows can disappear after them:

CREATE TABLE catalogs (
    catalog_id INT PRIMARY KEY,
    catalog_name VARCHAR(100) NOT NULL
);

CREATE TABLE products (
    product_id INT PRIMARY KEY,
    catalog_id INT NOT NULL,
    product_name VARCHAR(100) NOT NULL,
    CONSTRAINT fk_products_catalog
        FOREIGN KEY (catalog_id)
        REFERENCES catalogs (catalog_id)
        ON DELETE CASCADE
);

CREATE TABLE inventory_events (
    event_id INT PRIMARY KEY,
    product_id INT NOT NULL,
    event_type VARCHAR(40) NOT NULL,
    CONSTRAINT fk_inventory_events_product
        FOREIGN KEY (product_id)
        REFERENCES products (product_id)
        ON DELETE CASCADE
);

With that schema in place, the top-level delete can trigger changes in both child tables:

DELETE FROM catalogs
WHERE catalog_id = 300;

catalogs is the only table named in the DELETE, but matching rows in products and inventory_events can be removed as part of that same action. That is what makes cascades useful for rows that exist only as dependent parts of a larger row group.

Not every child row belongs in that category, though. Product rows may be tightly tied to a catalog, while payment rows, audit rows, archived records, or legal retention rows may need a different rule. Trouble starts when the same delete behavior is attached to every relationship just because it feels convenient. When a schema grows, that habit can turn a single delete into a far larger data change than intended.

Depth also becomes part of the conversation. Some database engines place limits on how deeply cascades can nest, while others block certain cascade graphs entirely if the same table can be reached by more than one route. That means cascade behavior is not just about table meaning. Graph structure and database product rules enter the discussion too.

Update Cascades

Parent identifiers usually stay fixed for the life of the row, which is why ON UPDATE CASCADE appears less frequently than ON DELETE CASCADE. Still, there are valid cases for it. Natural identifiers, externally assigned codes, or reference values imported from another source can change, and a foreign-key action can keep child rows aligned without requiring manual updates in every related table.

Regional reference data is a good fit for that idea because business-facing codes sometimes get renamed while the row itself stays in place:

CREATE TABLE regions (
    region_code VARCHAR(20) PRIMARY KEY,
    region_name VARCHAR(100) NOT NULL
);

CREATE TABLE warehouses (
    warehouse_id INT PRIMARY KEY,
    region_code VARCHAR(20) NOT NULL,
    warehouse_name VARCHAR(100) NOT NULL,
    CONSTRAINT fk_warehouses_region
        FOREIGN KEY (region_code)
        REFERENCES regions (region_code)
        ON UPDATE CASCADE
);

Parent code changes can then flow into matching child rows:

UPDATE regions
SET region_code = 'UPPER_MIDWEST'
WHERE region_code = 'MIDWEST';

With ON UPDATE CASCADE in place, rows in warehouses that still point to MIDWEST move to UPPER_MIDWEST during that same statement. Without it, the update is usually blocked if child rows still refer to the old value.

That product gap is one reason referenced identifiers are frequently kept stable when portability is part of the schema plan.

Checking rules also differ from one engine to another. Some systems separate NO ACTION from RESTRICT in deferred constraint cases, while others treat them as the same or check foreign keys immediately row by row during larger statements. Those differences do not always show up in a tiny example, but they can affect larger updates in noticeable ways.

SQL Server adds one more guardrail, it blocks foreign-key definitions that would let the same table appear more than one time in the set of cascading actions started by a single delete or update. That rule keeps the engine from having to pick between competing routes to the same destination table.

Indexing Foreign Keys

Delete speed and update speed can change sharply when child reference columns have no supporting index. Removing a parent row or changing a referenced value means the database must go find every matching child row before it can delete, block, or modify anything further down the chain. If that child column has no index, the engine may be pushed into broad scans or heavier locking than expected.

Store-level sales tables make that easier to read:

CREATE TABLE stores (
    store_id INT PRIMARY KEY,
    store_name VARCHAR(100) NOT NULL
);

CREATE TABLE sales_receipts (
    receipt_id BIGINT PRIMARY KEY,
    store_id INT NOT NULL,
    sale_total DECIMAL(10, 2) NOT NULL,
    CONSTRAINT fk_sales_receipts_store
        FOREIGN KEY (store_id)
        REFERENCES stores (store_id)
        ON DELETE CASCADE
);

CREATE INDEX idx_sales_receipts_store_id
    ON sales_receipts (store_id);

That index on sales_receipts.store_id gives the database a faster route to the child rows tied to a store. Without it, deleting a single store row can force the engine to search through a much larger portion of the receipt table before it finishes the cascade.

Confusion shows up here because the foreign-key definition itself can look complete. Parent columns already have index backing through a PRIMARY KEY or unique constraint, so the relationship appears fully covered at first glance. Child columns are different. In several major databases, the child-side index is a separate schema choice rather than something created automatically by the foreign-key declaration.

Indexing becomes more valuable as child tables grow. Small development datasets can hide the difference for quite a while, then larger tables make parent deletes or updates feel unexpectedly heavy. What looked fine with a few rows can turn into long scans and lock pressure after the data set grows.

Preventing Surprise Deletes

Unexpected deletes usually begin with a schema choice, not with the DELETE statement itself. Rows that are truly owned by the parent are strong candidates for CASCADE. Rows that stand on their own usually belong with RESTRICT, NO ACTION, or in some cases SET NULL. The safest direction comes from asking what the child row means after the parent row is gone. If it still has a valid independent purpose, automatic deletion may be the wrong rule. Larger schemas benefit from a dependency map before cascade actions go live. Trace the foreign-key graph table by table and follow what a top-level delete can touch. That review gets more important as the schema adds branch points, optional relationships, and multiple routes between related tables. Without that review, a delete that looks narrow in SQL text can remove far more data than expected.

Transactional testing is a practical way to inspect delete behavior before making it part of normal operations. Start a transaction, delete the parent row, review the affected child tables, and roll everything back after the inspection:

BEGIN;

DELETE FROM stores
WHERE store_id = 15;

SELECT *
FROM sales_receipts
WHERE store_id = 15;

ROLLBACK;

That kind of check gives a direct view of what the cascade touches without leaving the delete in place. It is also useful when several tables sit below the same parent and you want to verify that every child relationship behaves the way the schema author intended.

Keeping referenced identifiers stable avoids portability friction and keeps delete behavior as the main moving part in the foreign-key relationship.

Conclusion

Foreign key cascades give the database a built-in way to carry parent-child rules through deletes and updates without leaving related rows behind in an invalid state. After the relationship is defined, actions such as CASCADE, RESTRICT, NO ACTION, and SET NULL tell the engine how child rows should react, while schema depth, product differences, and child-side indexing affect how far those changes travel and how the database handles them. Read from that mechanical side, cascades are really about putting row behavior into the schema itself so related data reacts in a predictable way when parent data changes.

1. PostgreSQL Foreign Keys and Constraints

2. PostgreSQL Foreign Key Tutorial

3. MySQL Foreign Key Reference

4. SQL Server Foreign Key Relationships

5. SQL Server ALTER TABLE Constraint Reference

6. Oracle Database Data Integrity Guide

Share Alexander Obregon's Substack

Derived values can cut down on repeated expressions in day-to-day queries. In MySQL, this feature is called a generated column. For SQL Server, the comparable idea is called a computed column. Both let the database calculate a value from other columns in the same row, which keeps that logic attached to the table instead of repeating it across query text or application code. The main decisions come down to where that value lives, how the engine keeps it current, and what changes after an index is built on top of it. Those choices affect storage space, write overhead, and how much calculation still takes place during reads.

Subscribe now

Core Mechanics

Table definitions can hold derived columns, not just values entered directly. With that arrangement, the database keeps a formula at the schema level and applies it row by row under the rules of the engine. Query text stays shorter because the same expression does not need to be written again for every statement that touches the table. That gives the schema a central place for row-level calculations and makes the column part of the table itself rather than something rebuilt in scattered query text.

MySQL Generated Columns

Within MySQL, a generated column is declared with GENERATED ALWAYS AS (expression). The column still has a name and data type, but its value comes from the expression instead of direct input. MySQL supports two forms, VIRTUAL and STORED. A virtual generated column is calculated from other column values in the row, while a stored generated column is materialized as part of the row data. That choice is written directly into the column definition, so the table schema states how MySQL should treat the derived value from the start.

We can see this in action with this syntax example:

CREATE TABLE employees (
    employee_id BIGINT PRIMARY KEY,
    first_name VARCHAR(50) NOT NULL,
    last_name VARCHAR(50) NOT NULL,
    full_name VARCHAR(101)
        GENERATED ALWAYS AS (CONCAT(first_name, ' ', last_name)) VIRTUAL
);

full_name is derived from first_name and last_name, so the application does not send a separate value for it. Every row gets the same formula, and every query that reads full_name is reading from that single schema rule rather than rebuilding the concatenation in ad hoc SQL. That keeps the table definition in charge of the expression and cuts down on repeated formula text in queries.

Generated columns are also helpful when part of a stored value needs to be pulled into its own column. JSON data is a common case because queries can become noisy when the same extraction expression is repeated again and again. Moving that extraction into a generated column keeps the table definition more readable and gives the derived value a normal column name.

CREATE TABLE shipments (
    shipment_id BIGINT PRIMARY KEY,
    payload JSON NOT NULL,
    postal_code VARCHAR(20)
        GENERATED ALWAYS AS (
            JSON_UNQUOTE(JSON_EXTRACT(payload, '$.postalCode'))
        ) STORED
);

postal_code now comes from the payload document, but it behaves like a named column in the table definition. Queries can reference postal_code directly instead of carrying the full JSON_EXTRACT expression every time that value is needed. That is a practical reason generated columns show up in table schemas that carry semi-structured data.

MySQL treats generated columns as part of table structure, not as a loose display feature. That means they interact with schema rules such as indexing support, partitioning support, and foreign key restrictions in documented ways. At this stage, the main point is that a generated column is built into the table definition itself. It is not a query alias, not a view-only expression, and not a temporary value that exists only during a single SELECT.

MySQL also lets you add a generated column later through ALTER TABLE, which is useful when a table already exists and a repeated expression needs to move into the schema:

ALTER TABLE employees
ADD email VARCHAR(255) NOT NULL,
ADD email_domain VARCHAR(255)
    GENERATED ALWAYS AS (SUBSTRING_INDEX(email, '@', -1)) VIRTUAL;

That statement adds email_domain as a derived column without turning it into a value users type in directly. The table now carries the expression as part of its definition, so future queries can refer to email_domain as a column name instead of repeating the string function.

SQL Server Computed Columns

For SQL Server, the matching feature is a computed column. The table definition uses AS (expression) to tell SQL Server that the column value comes from an expression tied to other columns in the same row. By default, a computed column is virtual. SQL Server calculates it when the column is referenced instead of storing it physically in the table. If physical storage is wanted, PERSISTED can be added to the definition.

Let’s see how that looks:

CREATE TABLE Sales.Payroll (
    PayrollId int NOT NULL PRIMARY KEY,
    HoursWorked decimal(9,2) NOT NULL,
    HourlyRate decimal(9,2) NOT NULL,
    GrossPay AS (HoursWorked * HourlyRate)
);

GrossPay is derived from HoursWorked and HourlyRate, so there is no separate insert value for it. SQL Server keeps the formula with the table definition and applies that formula whenever GrossPay is referenced. That keeps arithmetic out of repeated query text and gives the schema a fixed rule for that derived value.

Data type behavior is part of the mechanics too. SQL Server resolves the computed column data type through its normal type precedence rules. If an expression combines values in a way that does not convert cleanly, the definition can fail or produce a type you did not expect. In those cases, CAST or CONVERT is used to state the intended result type directly. That becomes very helpful with money values, formatted strings, or expressions that combine integers with decimals.

This example shows that type choice inside the expression can be stated directly:

CREATE TABLE Sales.Invoices (
    InvoiceId int NOT NULL PRIMARY KEY,
    Subtotal decimal(10,2) NOT NULL,
    TaxRate decimal(5,4) NOT NULL,
    TaxAmount AS (CAST(Subtotal * TaxRate AS decimal(10,2)))
);

TaxAmount is still derived, but the expression now fixes the result type instead of leaving the final type entirely to implicit conversion rules. That keeps the schema aligned with the type you want the column to expose.

Physical storage is optional in SQL Server, and PERSISTED is the keyword that changes the column from virtual to stored in the table. A computed column marked that way is updated by SQL Server when the source columns change.

ALTER TABLE Sales.Payroll
ADD MonthlyPay AS (
    CAST(HoursWorked * HourlyRate * 4.33 AS decimal(12,2))
) PERSISTED;

MonthlyPay is now part of the stored row data rather than a value SQL Server recalculates only at read time. That storage choice becomes more important later when read cost and indexing come into view, but the mechanical point belongs here because PERSISTED is written right into the column definition.

Computed columns also follow stricter schema rules than regular columns. SQL Server does not treat them as ordinary input columns, so you do not insert or update them directly. They cannot be used as DEFAULT definitions, and they have documented rules around nullability, foreign keys, determinism, and indexing. Those rules make more sense once you remember what the column really is. Its value is tied to an expression owned by the table schema, not to direct user-supplied data.

Persistence Indexing Performance Tradeoffs

Storage choices change what the engine has to do during reads and writes. Virtual definitions leave the derived value out of the stored row, stored forms place it in row data, and indexes can place that value inside a search structure the optimizer can read directly. Read speed, row size, and update cost all move when that choice changes. That is where generated columns in MySQL and computed columns in SQL Server stop being only table-definition features and start affecting daily query behavior.

Read activity is only part of the story. Every derived value has to come from somewhere, and the database has to decide when that expression is evaluated, where the result is kept, and what extra maintenance follows when base columns change. Storage stays lower when the value is left virtual, but reads still have to evaluate the expression. Storing the result removes that repeated calculation for the column value itself, though row growth and update overhead rise in exchange. When an index is added, the trade becomes even more noticeable because the engine can search the derived value directly instead of recalculating it during a scan.

Leaving the Value Virtual

Virtual definitions leave the derived value outside the base row. MySQL handles that with VIRTUAL, while SQL Server leaves a computed column virtual until PERSISTED is added. In both products, the formula stays in the schema and the result is calculated when the column is referenced. That keeps row storage lower, which can be useful when the expression is short and the derived value is not pulled into every query.

Let’s look at a lightweight MySQL case:

CREATE TABLE page_views (
    view_id BIGINT PRIMARY KEY,
    viewed_at DATETIME NOT NULL,
    viewed_month CHAR(7)
        GENERATED ALWAYS AS (DATE_FORMAT(viewed_at, '%Y-%m')) VIRTUAL
);

viewed_month does not take space in the row as stored data. MySQL derives it from viewed_at whenever the column is needed. For a short date-format expression like this, that can be a reasonable trade because the source value already exists and the derived value is just a small transformation of it.

This form also helps keep queries from repeating the same expression again and again. Instead of writing DATE_FORMAT(viewed_at, '%Y-%m') in every report query, the table exposes viewed_month as a named column. That keeps query text shorter and reduces the chance that two queries end up with slightly different formulas for what should be the same value.

Read cost still exists, though. If a query touches a large number of rows and needs the virtual column for each row, the engine has to evaluate that expression across the read set. For light expressions that may be perfectly fine. For heavy expressions or very busy read paths, that repeated calculation can become more noticeable. Virtual form, then, fits best when lower row storage is more valuable than pre-saving the result.

SQL Server follows the same broad idea with different terminology. A computed column stays virtual by default, so SQL Server calculates it when the column is referenced instead of storing it physically in the row. That makes virtual computed columns a reasonable fit for columns that tidy up query logic but are not central to the busiest filters and result sets.

Storing the Value in the Row

Stored forms place the derived result inside the row itself. MySQL does that with STORED. SQL Server does it with PERSISTED. Both products then refresh the saved value when dependent columns change. Reads no longer need to recalculate the column value just to return it, but inserts and updates now carry extra maintenance and extra storage.

We can look at a short SQL Server definition that shows that change directly:

CREATE TABLE Sales.SubscriptionBilling (
    BillingId int NOT NULL PRIMARY KEY,
    MonthlyRate decimal(10,2) NOT NULL,
    BillableMonths int NOT NULL,
    ContractTotal AS (CAST(MonthlyRate * BillableMonths AS decimal(12,2))) PERSISTED
);

ContractTotal is stored with the row, so SQL Server updates it when MonthlyRate or BillableMonths changes. That can fit repeated reads of the same amount, mainly when the value appears in result sets again and again and there is little reason to recalculate it every time a query returns it.

MySQL presents the same trade with STORED generated columns. The derived value becomes part of row data, which means a read can fetch it directly rather than evaluating the expression for that column at that moment. That can be useful for values such as extended prices, tax totals, extracted JSON fields referenced constantly, or formatted values that appear across a broad slice of read traffic. Write activity becomes more expensive at the same time. If the dependent columns change, the saved derived value has to be refreshed. That refresh is automatic, but it still means more row maintenance during INSERT and UPDATE. Row size also grows because the result is now part of stored data rather than something built only when needed.

Stored form is really a trade between repeating CPU calculation during reads and carrying extra bytes and maintenance during writes. Tables that are read far more than they are changed may benefit from that trade. Tables with heavy write traffic and only occasional reference to the derived value may not gain much from storing it.

Indexing the Derived Column

Indexes can turn a derived column from a convenience feature into a search-friendly value the optimizer can use for filtering, ordering, and in selected cases index-only retrieval. MySQL allows indexes on stored generated columns, and InnoDB also supports secondary indexes on virtual generated columns. SQL Server allows indexes on computed columns too, but the expression must satisfy documented rules around determinism and precision, or be deterministic and persisted.

Let’s look at a MySQL table where a derived value comes out of JSON:

CREATE TABLE customer_events (
    event_id BIGINT PRIMARY KEY,
    payload JSON NOT NULL,
    region_code VARCHAR(8)
        GENERATED ALWAYS AS (
            JSON_UNQUOTE(JSON_EXTRACT(payload, '$.region'))
        ) VIRTUAL,
    INDEX ix_region_code (region_code)
);

The base row does not store region_code, yet the secondary index does store the generated value in its index records. That gives MySQL something searchable without requiring the table itself to carry the derived value as stored row data. Queries that filter by region_code can then use the index instead of recalculating the JSON_EXTRACT expression across a scan of the table. Also, if a query can be satisfied from the secondary index alone, MySQL can read the generated value from the index record rather than recalculating it from the base row. That makes indexed virtual generated columns more useful than they first appear, because the value may still live in searchable storage even though it is not part of the stored row itself.

SQL Server handles indexed computed columns with stricter rules. Determinism is part of that rule set, which means the expression must always return the same result for the same input values. Precision also matters. If a computed expression is deterministic and precise, indexing can be allowed without storage in some cases. If the expression is deterministic but not precise, PERSISTED becomes part of the route to indexing.

This definition shows a common SQL Server case:

CREATE TABLE Sales.InvoiceLines (
    InvoiceLineId int NOT NULL PRIMARY KEY,
    Quantity int NOT NULL,
    UnitPrice decimal(10,2) NOT NULL,
    ExtendedAmount AS (CAST(Quantity * UnitPrice AS decimal(12,2))) PERSISTED
);

CREATE INDEX IX_InvoiceLines_ExtendedAmount
    ON Sales.InvoiceLines (ExtendedAmount);

ExtendedAmount is now both stored and indexed, which gives SQL Server a searchable derived amount instead of forcing a fresh multiplication across rows during every relevant filter or sort. That can help range predicates, ordered retrieval, and some join cases where the computed value is central to the query.

Constraints can follow those same indexing rules in SQL Server. A computed column may participate in PRIMARY KEY or UNIQUE constraints if the expression satisfies the documented requirements. That matters because it shows the engine is treating the derived value as more than a display convenience. Once the expression meets the rule set, the column can become part of schema-level uniqueness and search structures.

Write overhead rises here too. Each index on a derived column has to be maintained when the dependent values change. In MySQL that means generated values are materialized into secondary index records during row changes. In SQL Server the indexed computed value must also stay in sync with updates to the underlying columns. Reads can get much faster, but that speed comes with added write cost and extra storage in index structures.

Picking the Right Form

Choice comes down to query behavior, row size, expression cost, and indexing rules in the database engine in front of you. Virtual or nonpersisted form fits best when the formula is light, the derived value mainly keeps query text shorter, and storing the result inside every row would add little value. Stored or persisted form fits better when the same derived value is returned constantly or when storage in the row lines up better with the indexing rules needed for that column.

In MySQL, a table that keeps base rows lean while still exposing a searchable derived value shows that balance well:

CREATE TABLE support_requests (
    request_id BIGINT PRIMARY KEY,
    payload JSON NOT NULL,
    priority_code VARCHAR(10)
        GENERATED ALWAYS AS (
            JSON_UNQUOTE(JSON_EXTRACT(payload, '$.priority'))
        ) VIRTUAL,
    INDEX ix_priority_code (priority_code)
);

priority_code stays out of the base row, but the secondary index still carries the derived value for search use. That can be a good fit when preserving row space is more important than storing the extracted value inside every table record, yet fast filtering on that derived field is still needed.

SQL Server makes the choice a little tighter because indexing rules are more restrictive. If an expression is deterministic and precise, an indexed computed column may be allowed without persistence. If the expression is deterministic but not precise, PERSISTED can become necessary before the column is indexed. That means the storage decision is tied not only to read frequency, but also to the rule set attached to computed-column indexing.

Read-heavy reporting, repeated range filters, and frequent ordering by the derived value all push the decision toward indexed forms. Tables with heavy write traffic and only occasional reference to the derived value usually fit better with leaner definitions. Good schema choices come from reading the query load first, then matching that load to row growth, update overhead, and the indexing limits of the engine involved.

Conclusion

Generated columns in MySQL and computed columns in SQL Server give the table a built-in way to derive values from other columns, which moves repeated formulas out of day-to-day queries and into the schema itself. From there, the mechanics come down to when the value is calculated, where it is kept, and how that choice affects reads, writes, and indexing. Virtual definitions save row space but keep calculation on the read side, while stored or persisted definitions add row maintenance in exchange for direct access to the derived value itself. After indexing enters the discussion, those derived values can also become searchable parts of the table structure, which is why the best fit comes from how the expression is read, updated, and filtered in the database.

1. MySQL Generated Columns

2. MySQL Secondary Indexes on Generated Columns

3. MySQL JSON Function Reference

4. SQL Server Computed Columns

5. SQL Server CREATE INDEX

6. SQL Server ALTER TABLE computed_column_definition

Share Alexander Obregon's Substack

Good partition pruning begins during query planning, when the database decides which partitions it can skip before row reads start. That only happens when the optimizer can connect the WHERE clause to the stored partition boundaries tightly enough to rule partitions out ahead of time or while the statement is running. Direct comparisons on the partition column usually give it that proof. Put the column inside a function, compare it through the wrong data type, or write the filter in a form the engine cannot fold back to those boundaries, and a query that should touch a small slice of data may end up scanning every partition. Across PostgreSQL, MySQL, BigQuery, Oracle Database, and SQL Server, that broad rule stays the same, but the exact pruning behavior changes from one product to the next.

Subscribe now

How Partition Pruning Works

Before row reads begin, the optimizer tries to narrow the table down to the partitions that could still hold matching data. That step happens by comparing the filter in the query with the partition map stored for the table. If the filter lines up with those boundaries, the engine can skip partitions before it spends time reading them. PostgreSQL explains pruning as proving that a partition cannot contain rows that satisfy the WHERE clause, while Oracle talks about building a partition access list from the FROM and WHERE clauses.

Reading Partition Boundaries

Beneath a partitioned table, the rows live in separate physical pieces even though SQL still treats the object as a single table. Range partitioning splits values into intervals such as one month per partition. List partitioning assigns named values to specific partitions. Hash partitioning routes rows into buckets from a hash result. That layout gives the optimizer a map it can reason from before the scan begins.

Monthly date partitioning helps make that easier to see. January rows can live in one partition, February rows in the next, and March rows in the next after that. With that layout in place, a query asking only for March data does not need to open January or February. The planner can compare the filter to the stored bounds and narrow the scan to the partitions whose ranges still overlap the requested dates.

CREATE TABLE meter_readings (
    account_id BIGINT NOT NULL,
    reading_date DATE NOT NULL,
    kilowatt_hours NUMERIC(10,2) NOT NULL
) PARTITION BY RANGE (reading_date);

CREATE TABLE meter_readings_2026_01
    PARTITION OF meter_readings
    FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');

CREATE TABLE meter_readings_2026_02
    PARTITION OF meter_readings
    FOR VALUES FROM ('2026-02-01') TO ('2026-03-01');

CREATE TABLE meter_readings_2026_03
    PARTITION OF meter_readings
    FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');

That definition stores three separate date intervals for reading_date. When the optimizer sees a filter that falls fully inside March, only the March partition overlaps that request. January and February can be ruled out before row filtering begins. The same general idea applies to list partitions and hash partitions, but the stored map looks different. List partitions store named values or value groups. Hash partitions store bucket assignments rather than human-readable ranges. Pruning also needs to be separated from indexing. An index can help locate rows inside a chosen partition, but partition pruning happens earlier. The pruning step asks which partitions still belong in the scan at all. After that decision is made, indexes inside those remaining partitions can still affect how rows are read. That distinction helps explain why a table can prune partitions even when the partition column itself has no index.

Filters That Map to Boundaries

Most pruning-friendly filters leave the partition column visible and compare it to values that line up with the partition map. Equality predicates do that well. IN lists can do it well too when the listed values map to a small set of partitions. Range predicates are also strong candidates when the start and end values match the stored boundaries closely enough for the optimizer to narrow the scan early.

Half-open date ranges are a common form because they line up neatly with range partition boundaries and avoid overlap between adjacent time windows:

SELECT account_id, kilowatt_hours
FROM meter_readings
WHERE reading_date >= DATE '2026-03-01'
  AND reading_date < DATE '2026-04-01';

That filter leaves reading_date alone and compares it directly to typed boundary values. March rows can match. January, February, and April rows cannot. With monthly partitions, the optimizer can tie that predicate to the March partition and avoid scanning the others. Date filtering written this way also avoids boundary confusion that can happen when people try to express month filters with a single ending date that still includes time values.

Equality predicates are even tighter when the partition layout matches them. List partitioning is a good example. If a table is partitioned by a status column, the optimizer can look at the requested values and narrow the scan to the partitions that own those statuses:

SELECT ticket_id, opened_at
FROM support_tickets
WHERE ticket_status IN ('OPEN', 'PENDING');

With list partitioning, those values are already part of the partition map. If both values belong to one partition, the scan can stay there. If they belong to two partitions, the engine only needs those two. Nothing outside those mapped values needs to be touched.

BigQuery follows the same broad rule, but ingestion-time partitioned tables use the _PARTITIONTIME pseudocolumn for pruning, and daily ingestion-time tables also expose _PARTITIONDATE:

SELECT order_id, total_amount
FROM dataset.orders
WHERE _PARTITIONDATE BETWEEN DATE '2026-03-01' AND DATE '2026-03-07';

That filter narrows the scan to the daily partitions covering those dates. The pruning idea is still the same. The optimizer needs a direct relationship between the filter and the partition map. When that relationship is visible in the predicate, the engine can reduce how much storage it reads before row filtering moves forward.

Not every partitioning method uses calendar-style ranges, but the same planning rule still applies. Integer-range partitioning narrows scans from numeric boundaries. List partitioning narrows scans from stored value groups. Hash partitioning is less intuitive to read by eye, yet it still rests on the same principle. The optimizer has to map the filter back to the partition definitions closely enough to tell which partitions can still contain matching rows.

PostgreSQL adds a useful characteristic to that broad idea, partition pruning does not have to happen only at initial plan creation. It can also happen during execution, which means partition selection can still narrow later in the statement lifecycle when the database has the needed values. The basic requirement stays the same throughout. The filter has to give the database enough information to connect the request back to partition boundaries.

Why Some Filters Miss Pruning

Pruning depends on how much the optimizer can prove from a predicate before or during partition selection. That proof gets harder when the partition column is wrapped in extra logic, compared through an implicit conversion, or fed by values that arrive later than the planner can reason about. Row filtering can still return the right result, but the database may lose its chance to cut away large parts of the table early, which pulls more partitions into the scan.

Functions Over the Partition Column

Putting a function around the partition column is a common way pruning gets lost. Oracle calls this out for transformations such as CAST and TRUNC, and BigQuery says function-based filters over an integer-range partition column do not prune partitions. After the column is wrapped, the optimizer no longer sees the stored boundary compared against a value in its original form. It first has to reason through the added expression before it can map the predicate back to partition metadata, and some engines stop there.

Take a month filter written like this:

SELECT order_id, order_total
FROM sales_orders
WHERE DATE_TRUNC('month', order_date) = DATE '2026-06-01';

That query can return the right rows, but the month calculation is applied to order_date instead of leaving order_date exposed for boundary matching. A range written directly on the column gives the optimizer a shorter route back to the stored bounds:

SELECT order_id, order_total
FROM sales_orders
WHERE order_date >= DATE '2026-06-01'
  AND order_date < DATE '2026-07-01';

Same month window, very different pruning odds. Predicates written directly on the column match stored bounds more closely, so the planner has a better shot at cutting away unneeded partitions before scan time.

BigQuery calls out the same issue for integer-range partitions with arithmetic on the partition column:

SELECT *
FROM dataset.customer_events
WHERE customer_id + 1 BETWEEN 30 AND 50;

That arithmetic changes the pruning picture before the scan starts. A rewritten filter such as customer_id BETWEEN 29 AND 49 puts the partition column back in direct view. MySQL is less rigid in a few date partition layouts. If the partition expression itself is based on YEAR(), TO_DAYS(), or TO_SECONDS(), pruning can still happen because MySQL knows how to map date predicates back through those specific expressions. Blanket rules about functions hide that product-level detail, so the real question is narrower. Can the engine translate this predicate into partition-boundary choices for this table definition.

Casts That Force Conversions

Type conversions can block pruning without looking dramatic in the SQL text. Oracle separates static pruning from dynamic pruning and warns that data type conversions usually move a statement away from static pruning. With DATE columns, it also notes that implicit conversion depends on session NLS settings, while a proper TO_DATE call lets the database identify the intended date value more precisely.

Take an Oracle-style filter written against a DATE partition column:

SELECT shipment_id, delivered_on
FROM shipments
WHERE delivered_on >= TO_DATE('2026-06-15', 'YYYY-MM-DD')
  AND delivered_on < TO_DATE('2026-06-16', 'YYYY-MM-DD');

The conversion stays on the constant side there, which keeps the partition column in its native type. On date-partitioned tables, that small text choice can separate compile-time pruning from a later pruning decision that has less room to narrow the scan early. Trouble starts when the partition column itself is cast or when a string literal is left for the database to interpret through session formatting rules. Both forms can return valid rows, yet they make boundary matching less direct.

Runtime Values Across Engines

Database products do not all treat runtime values the same way. PostgreSQL can prune during execution, not just during planning, and its docs name prepared statement parameters, subquery values, and parameterized nested loop values as inputs that can still remove partitions after planning has started. Oracle has the same static-versus-dynamic split and lists bind variables, subqueries, and nested loop joins as dynamic pruning cases.

BigQuery takes a stricter route for partition filters that are meant to limit scanned partitions. Seeing a filter tied to a subquery result is a good example:

SELECT *
FROM dataset.events
WHERE _PARTITIONTIME = (
    SELECT MAX(bookmark_ts)
    FROM dataset.event_bookmarks
);

That statement ties _PARTITIONTIME to a subquery result instead of a constant boundary. In BigQuery, filters written that way do not limit scanned partitions the same way a constant partition filter does. Differences like that become important when the same application SQL is expected to behave similarly across products. Prepared or derived values that still prune in PostgreSQL or Oracle can leave BigQuery reading far more data.

What to Check in Execution Plans

Execution plans are where pruning has to prove itself. Reading the WHERE clause alone is not enough. PostgreSQL gives two different clues depending on timing. During initialization, EXPLAIN can report Subplans Removed. During execution, EXPLAIN ANALYZE can reveal partition subplans with different loops counts, and some partitions can appear as (never executed) when runtime pruning removes them every time.

Take this PostgreSQL plan check:

EXPLAIN ANALYZE
SELECT invoice_id
FROM monthly_invoices
WHERE invoice_date >= DATE '2026-06-01'
  AND invoice_date < DATE '2026-07-01';

Plan output tells the real story. If pruning happened during initialization, Subplans Removed reports partitions taken out before execution moved forward. If pruning happened later, partition subplans can carry different loops counts, and some can appear as (never executed).

Oracle exposes pruning very directly through PSTART and PSTOP. When the plan reports PARTITION RANGE SINGLE, those columns identify the accessed partition span. When it reports PARTITION RANGE ALL, pruning did not narrow the scan. Dynamic pruning can show KEY markers in PSTART and PSTOP rather than fixed partition numbers resolved at parse time.

SQL Server and BigQuery surface the same question in different ways. SQL Server focuses on partition elimination in its plan tools, while BigQuery exposes the result through bytes scanned and bytes processed estimates, including dry runs. If a filter was written to touch a narrow date slice but the plan or estimate still reflects the full table, pruning did not happen in the way the query text implied.

Conclusion

Partition pruning comes down to how directly a filter points the optimizer to partition boundaries. Keep the partition column visible, compare it in its native type, and write predicates in forms the engine can map back to stored ranges or values. If that link gets blurred by functions, conversions, or late-bound expressions, the database has less room to cut the scan down early, which is why two queries that return the same rows can read very different amounts of data.

1. PostgreSQL Declarative Partitioning

2. PostgreSQL Using EXPLAIN

3. MySQL Partition Pruning

4. BigQuery Query Partitioned Tables

5. Oracle Database Partition Pruning

6. SQL Server Partitioned Tables and Indexes

Share Alexander Obregon's Substack

Bulk updates usually start with a business rule, not a cursor. Maybe a customer tier needs to change, a balance needs a refresh, a summary column needs values from detail rows, or a shipment flag has to flip after related data arrives. That is where row by row logic creeps in, with one row read, one result calculated, and one row updated again and again. SQL gives you a better route. PostgreSQL, SQL Server, and MySQL all support joined updates and CTE-based updates, so the database can evaluate the target rows, source rows, and filter rules in one statement instead of repeating the same update row by row.

Subscribe now

Why Set Based Updates Fit SQL

Set based thinking lines up well with how an UPDATE statement is built. SQL starts from a group of rows that match a rule, then assigns new values across that full row set in a single statement. PostgreSQL supports UPDATE with a FROM clause and RETURNING, SQL Server supports UPDATE with FROM, CTEs, and OUTPUT, and MySQL 8.4 supports UPDATE statements that can begin with WITH along with multi-table updates. Those features let the database read the target rows, source rows, and filter rules as one unit instead of repeating the same write row by row.

SQL Wants the Full Change Set

Set based SQL begins with the rule that defines the rows to change. The statement is not focused on which row comes next. Its focus stays on which rows belong in the update and what value each of those rows should receive. That lines up well with SQL because the full row set is visible from the start.

Reading an update through that lens makes the structure much easier to follow. Start with the WHERE clause. That is where the target row set is defined. Then move to the SET clause. That is where the new value is defined for every row that passed the filter. Row by row code breaks those ideas apart and repeats them again and again. Set based SQL keeps the rule in one place, which makes it much easier to read against the business change you are trying to make.

Take a small status change as a first example:

UPDATE support_ticket
SET priority = 'HIGH'
WHERE status = 'OPEN'
  AND days_open >= 30;

Nothing in that statement depends on processing rows one at a time. The filter picks every qualifying row, and the assignment applies the same new value across that full set.

That same idea still holds when the new value depends on data already stored in the row. SQL can evaluate the rule for every qualifying row inside the statement itself, so there is no need to pull rows into a loop just to classify them and send them back.

UPDATE support_ticket
SET priority =
    CASE
        WHEN days_open >= 30 THEN 'HIGH'
        WHEN days_open >= 7 THEN 'MEDIUM'
        ELSE 'LOW'
    END
WHERE status = 'OPEN';

CASE keeps the decision logic inside the update, which lets the database evaluate the full row set under one statement. That is a better fit for SQL than reading a row, calculating a label outside the database, writing it back, and repeating the cycle until the table is done.

CTEs fit naturally into that same idea because sometimes the row set needs a name before the write happens. Sometimes the filter logic is long enough that it reads better when it is separated from the final assignment. A CTE gives you a statement-scoped result set that can feed the update without turning the operation into row by row code.

Take a look at this form:

WITH eligible_ticket AS (
    SELECT ticket_id, days_open
    FROM support_ticket
    WHERE status = 'OPEN'
)
UPDATE support_ticket t
SET priority =
    CASE
        WHEN e.days_open >= 30 THEN 'HIGH'
        WHEN e.days_open >= 7 THEN 'MEDIUM'
        ELSE 'LOW'
    END
FROM eligible_ticket e
WHERE t.ticket_id = e.ticket_id;

The update still happens as a set. The CTE just gives the qualifying rows a name before the write takes place. That keeps the rule readable without changing the set based nature of the statement.

One Source Row for Each Target Row

Joined updates need one rule kept firmly in view. Each target row should line up with a single source row for the value being assigned. When that does not happen, the statement can produce results you did not intend because more than one source row is feeding the same target row. That idea comes into better focus with a table that stores summary values. Say an inventory table stores the latest sale time for each item. A sale_history table can hold several sales for the same item. Joining inventory straight to sale_history leaves one inventory row matching several source rows, which is not a good match for a joined update.

The safer route is to reduce the source side first so that only one row remains for each target row:

WITH latest_sale AS (
    SELECT
        item_id,
        MAX(sale_time) AS latest_sale_time
    FROM sale_history
    GROUP BY item_id
)
UPDATE inventory i
SET last_sale_at = ls.latest_sale_time
FROM latest_sale ls
WHERE i.item_id = ls.item_id;

That CTE changes the source side from several sale rows per item to a single row per item. After that reduction, each inventory row has one source row feeding the assigned value, which is exactly what a joined update needs.

The same rule shows up in smaller cases too. Think about an account table that stores the most recent payment date. Payment history can contain several rows for one account. Jumping straight into a joined update would leave the account row matching several payment rows, so the source side needs to be narrowed first.

This version keeps the join aligned with one source row for each account:

WITH latest_payment AS (
    SELECT
        account_id,
        MAX(payment_date) AS last_payment_date
    FROM payment_history
    GROUP BY account_id
)
UPDATE account a
SET last_payment_date = lp.last_payment_date
FROM latest_payment lp
WHERE a.account_id = lp.account_id;

Grouping, ranking, or some other row reduction step is what makes joined updates dependable in cases like these. After the source side has been narrowed to one row per target row, the final update reads much more naturally. You can look at the statement and see the target table, the source result, the join condition, and the assigned value without having to trace a loop outside SQL.

That difference is one of the biggest reasons set based updates fit SQL so well. Row by row code hides the matching rule inside repeated reads and writes. Keeping the full row relationship inside the statement makes it much simpler to spot when the source side is too wide and when it needs to be reduced before the write happens.

Rewriting Common Row by Row Updates

Row by row update logic usually repeats the same cycle. A row is read, a value is derived, that value is written back, and the database moves on to the next row. SQL can fold that repeated cycle into a single UPDATE when the new value comes from a lookup row, a grouped detail result, or a rule written as an expression inside the statement. PostgreSQL, SQL Server, and MySQL all support forms of that rewrite through joined updates and CTE-fed updates, and those same joins or CTEs can be turned into a preview query before data changes are written.

Moving from row by row logic to a set based statement starts with a small change in how the update is read. Instead of thinking about the next row to fetch, focus on the full target row set, the source of the new value, and the condition that limits the write. That keeps the update tied to the data rule itself rather than to a loop wrapped around it.

Copying Values From a Lookup Table

Lookup-driven updates are one of the first places where cursor logic starts to feel unnecessary. Code that fetches a target row, reads a related lookup row, copies a value, and repeats is really expressing a join. After that becomes visible, the rewrite is usually very natural. Put the target table in the UPDATE, join to the lookup source, filter the rows that need the change, and assign the lookup value in the SET clause. PostgreSQL and SQL Server support this through UPDATE with FROM, while MySQL supports the joined form through multi-table UPDATE syntax.

Take a postal-rate update where each open shipment should copy the current zone from a reference table:

UPDATE shipment s
SET zone_code = z.zone_code
FROM postal_zone z
WHERE s.postal_code = z.postal_code
  AND s.status = 'OPEN';

Nothing in that statement needs a cursor. shipment is the target, postal_zone is the lookup source, the join ties the two tables together, and the filter limits the write to open shipments. Putting that logic in a loop outside SQL would just repeat the same lookup again and again.

SQL Server reads in much the same way, with the target alias named in the UPDATE and the join written in the FROM clause:

UPDATE pr
SET tax_rate = tr.rate
FROM purchase_record AS pr
JOIN tax_rate AS tr
  ON pr.state_code = tr.state_code
WHERE pr.invoice_status = 'PENDING';

That rewrite fits any case where the new value already lives in a related table. Postal zones, tax rates, office codes, district names, supervisor IDs, and billing categories all fall into that group. What looked like row by row logic in application code usually turns out to be a joined update waiting to be written.

Filling Summary Columns From Detail Rows

Summary columns need an extra step before the update happens. Detail rows have to be reduced to a single row per parent first. Without that reduction, a joined update can match the same target row against several detail rows, which is not a good fit when the target table stores a single summary value. Aggregates such as SUM, MAX, MIN, and COUNT turn the detail side into a row set that can feed the update safely.

Take a shipping example where manifest.total_weight should reflect the sum of all package weights tied to that manifest. Cursor logic would read one manifest, sum its package rows, update the weight, and continue to the next manifest. SQL can do that in one statement by grouping the detail rows first and then joining the grouped result back to the target table:

WITH manifest_weight AS (
    SELECT
        package.manifest_id,
        SUM(package.weight_lbs) AS total_weight
    FROM package
    GROUP BY package.manifest_id
)
UPDATE manifest m
SET total_weight = mw.total_weight
FROM manifest_weight mw
WHERE m.manifest_id = mw.manifest_id;

That grouped CTE changes the source side before the update starts. Instead of several package rows for the same manifest, the update sees a single grouped row per manifest with the final weight already calculated. That lines up with the target column being filled. A single total_weight value belongs to each manifest row, so the source side should also present a single row for each manifest.

The same rewrite fits balances, totals, last activity timestamps, line counts, and similar parent-level values that come from detail tables. Repeated loop logic was never really about processing each parent in isolation. The actual rule was to reduce the detail rows to one result per parent and then write that result back.

Applying Rule Based Changes With a CTE

Rule-driven updates do not always need a second base table. Some loops exist only because the decision logic feels easier to write outside SQL. The loop reads a row, checks several columns, assigns a label or flag, and then updates that row. SQL can keep that rule inside the statement through CASE, and a CTE helps when the target row set or the intermediate values deserve a name before the final write.

Let’s see a subscription table where active rows need a renewal status based on days left and unpaid balance:

WITH renewal_target AS (
    SELECT
        subscription_id,
        days_to_expiry,
        unpaid_balance
    FROM subscription
    WHERE account_status = 'ACTIVE'
)
UPDATE subscription s
SET renewal_status =
    CASE
        WHEN r.unpaid_balance > 0 THEN 'HOLD'
        WHEN r.days_to_expiry <= 7 THEN 'RENEW_NOW'
        WHEN r.days_to_expiry <= 30 THEN 'REVIEW'
        ELSE 'OK'
    END
FROM renewal_target r
WHERE s.subscription_id = r.subscription_id;

That CTE does not change the fact that the update is still set based. Its job is to name the qualifying row set and keep the final UPDATE focused on assigning the new value. That can help when the filter logic is longer, when intermediate calculations belong in the source row set, or when the statement reads better after the row selection has been separated from the final assignment.

Keeping the rule inside SQL also keeps the logic near the data it depends on. That makes the rewrite from row loop to set update much easier to follow, because the filtering logic, the decision logic, and the write all stay in the same statement.

Previewing Rows Before the Write

Before any update runs, the safest habit is to turn the same join or CTE into a SELECT first. That preview tells you which rows qualify, what value is stored now, and what value is about to be written. Nothing about that step is separate from the final update. It is the same row logic with the write removed, which makes it a very useful checkpoint before data changes happen.

For example, say we are working with a membership-fee update where each plan fee should be refreshed from a rate table. Before the write, run the row-finding logic as a SELECT:

WITH target_fee AS (
    SELECT
        m.member_id,
        m.monthly_fee AS current_fee,
        r.monthly_fee AS new_fee
    FROM member_plan m
    JOIN fee_rate r
      ON m.plan_code = r.plan_code
    WHERE m.monthly_fee <> r.monthly_fee
)
SELECT
    member_id,
    current_fee,
    new_fee
FROM target_fee
ORDER BY member_id;

That preview makes the pending change visible before the update touches the table. You can inspect the candidate rows, check that the join is matching the right rate, and verify that the new value is what you expected. Turning an update into a preview SELECT like this is one of the best habits to build around set based writes.

After the write, PostgreSQL can return the affected rows directly from the UPDATE statement:

UPDATE member_plan m
SET monthly_fee = r.monthly_fee
FROM fee_rate r
WHERE m.plan_code = r.plan_code
  AND m.monthly_fee <> r.monthly_fee
RETURNING
    m.member_id,
    m.monthly_fee AS updated_fee;

SQL Server has a similar review step with OUTPUT, which can return the old row image from deleted and the new row image from inserted for each affected row. That is useful when you want a record of what changed right after the statement runs, but it does not replace the preview query. The preview still answers the question that comes first, which rows are about to change and what values they are about to receive.

Conclusion

Set based updates keep the row filter, value calculation, joins, grouping, and final write inside the same SQL statement, which is what makes them such a good replacement for cursors in jobs like lookup copies, summary refreshes, and rule-based status changes. After the source rows have been narrowed to the right grain and the update logic has been previewed with a matching SELECT, the final write becomes more natural to reason through, more reliable to check before it runs, and more closely aligned with the data rule being applied.

1. PostgreSQL UPDATE

2. PostgreSQL WITH Queries

3. PostgreSQL Returning Data From Modified Rows

4. SQL Server UPDATE Transact-SQL

5. SQL Server OUTPUT Clause

6. MySQL UPDATE Statement

7. MySQL WITH Common Table Expressions

Share Alexander Obregon's Substack

Summarizing large result sets is a normal part of reporting, dashboards, exports, finance work, and ad hoc query work. The hard part starts when one report needs detailed rows, subtotal rows at more than one level, and a grand total in the same result. Writing a separate GROUP BY query for every total level can get repetitive fast, and it also makes the SQL harder to read. SQL solves that with grouping extensions that let one query return several grouped levels at the same time. That is where GROUPING SETS, ROLLUP, and CUBE come in. They all return grouped output, but they do not produce the same structure. ROLLUP follows a hierarchy, CUBE returns every possible grouping combination, and GROUPING SETS lets you state the exact grouped levels you want. Major database engines support these features a little differently, so those differences matter when you want one query to work well across more than one system.

Subscribe now

How Multi Level Grouping Works in One Query

Grouped extensions build on the same foundation as plain GROUP BY, but they remove the limit of returning only one grouping level from a query block. Regular grouping divides rows by the selected columns and applies aggregate functions such as SUM, COUNT, or AVG to each group. GROUPING SETS, ROLLUP, and CUBE keep that same aggregate process, but they let the database return multiple grouped levels in one result. That change is what makes multi-level totals possible without stacking separate grouped queries.

Why Plain GROUP BY Stops Short

Plain GROUP BY works nicely when the report asks for a single grouping level and nothing more. Group order rows by sales_year and region, and the result gives a row per year and region pair. That answers a reporting question well, but it does not also return yearly totals by themselves or a full grand total.

This query gives a good starting point:

SELECT
    sales_year,
    region,
    SUM(net_amount) AS total_net_amount
FROM order_summary
GROUP BY sales_year, region
ORDER BY sales_year, region;

In that code, the query returns totals for each sales_year and region pair. If the report also needs a row per year across all regions, plain GROUP BY cannot add that second grouped level automatically. You would need a separate grouped query.

Repetition starts to grow at that point. The first grouped query answers the detailed level. The second answers the higher total. The third answers the grand total. The output can still be built, but the SQL now has to repeat the source table, repeat the filter logic, and repeat the aggregate expression. As subtotal levels grow, that repetition spreads across more of the statement.

In this kind of query we can see what that looks like:

SELECT
    sales_year,
    region,
    SUM(net_amount) AS total_net_amount
FROM order_summary
GROUP BY sales_year, region

UNION ALL

SELECT
    sales_year,
    NULL AS region,
    SUM(net_amount) AS total_net_amount
FROM order_summary
GROUP BY sales_year

UNION ALL

SELECT
    NULL AS sales_year,
    NULL AS region,
    SUM(net_amount) AS total_net_amount
FROM order_summary;

That works because each SELECT returns a single grouping level, and UNION ALL stacks those results into a single output. Still, the statement gets harder to scan because the grouping intent is spread across three query blocks instead of living in a single grouped clause. It also raises the chance of drift. If a filter is added to some branch but missed in another, the totals change.

Grouped extensions solve that by keeping the grouped levels in a single GROUP BY clause. The same source rows are still being summarized, but the grouping logic becomes more direct. Instead of saying run three different grouped queries and stack them, you say return these grouped levels from this grouped statement.

Something else worth mentioning is that plain GROUP BY always returns rows at a single granularity for that query block. It does not carry a built-in idea of subtotal rows. That is why subtotal features had to be added as separate syntax rather than folded into normal grouping behavior.

GROUPING SETS, ROLLUP and CUBE

The general idea behind all three features is the same, a grouped statement can return more than a single grouping level. The difference is in how those levels are chosen. GROUPING SETS is the most direct form. You name the grouping levels you want, and the database returns those levels and no others. That makes it a good fit when the report has a fixed subtotal layout.

This version asks for exactly three grouped levels:

SELECT
    department,
    job_title,
    SUM(base_salary) AS total_salary
FROM employee_payroll
GROUP BY GROUPING SETS
(
    (department, job_title),
    (department),
    ()
)
ORDER BY department, job_title;

Here, the query asks for three grouped levels. The first grouping set returns totals by department and job_title. The second returns totals by department alone. The empty grouping set () returns the grand total across the whole input. That empty set matters because it tells the database to aggregate all remaining rows into a final summary row.

ROLLUP is shorter when the grouped levels follow a hierarchy. Hierarchy means the grouped columns naturally move from more detailed to less detailed in left to right order. If a report groups by year, quarter, and month, that order already forms a ladder from detailed rows up to broader totals. ROLLUP follows that ladder by removing grouping columns from right to left.

Take a query like this:

SELECT
    sales_year,
    sales_quarter,
    sales_month,
    SUM(net_amount) AS total_net_amount
FROM monthly_sales
GROUP BY ROLLUP(sales_year, sales_quarter, sales_month)
ORDER BY sales_year, sales_quarter, sales_month;

The single ROLLUP clause expands into grouped levels that match the column order. It returns totals by year, quarter, and month, then year and quarter, then year, then the full grand total. Put plainly, the placement of columns inside ROLLUP matters. Swap them, and you change the subtotal path.

Left to right order deserves a little extra attention because it explains a lot of potential confusion. ROLLUP(region, store) produces store totals inside each region, then region totals, then the grand total. ROLLUP(store, region) produces region totals inside each store, which is a very different report. The function is not reading business meaning from the names. It is following column order. CUBE goes wider than ROLLUP. Instead of following a single hierarchy, it returns every possible grouping combination from the listed expressions. With two grouped columns, that means four grouped levels. With three grouped columns, that means eight grouped levels. Row counts can grow fast, which is why CUBE fits best when every subtotal combination is actually useful to the report.

We can see that wider grouping style here:

SELECT
    warehouse_region,
    shipping_method,
    SUM(package_count) AS total_packages
FROM shipment_facts
GROUP BY CUBE(warehouse_region, shipping_method)
ORDER BY warehouse_region, shipping_method;

That query returns totals for warehouse_region and shipping_method pairs, totals by warehouse_region, totals by shipping_method, and a grand total. Notice what makes CUBE different from ROLLUP. ROLLUP(warehouse_region, shipping_method) would not return totals by shipping_method alone. CUBE does, because it includes every grouping combination across the listed columns.

Small side by side comparisons can also make the distinction easier to see. These two queries look close, but they do not ask for the same grouped result:

SELECT
    campus,
    course_level,
    COUNT(*) AS course_count
FROM class_schedule
GROUP BY ROLLUP(campus, course_level);

That ROLLUP returns grouped rows for campus and course_level, then campus, then the grand total.

SELECT
    campus,
    course_level,
    COUNT(*) AS course_count
FROM class_schedule
GROUP BY CUBE(campus, course_level);

And this CUBE returns the same grouped rows as the rollup query, but it also adds totals by course_level alone across all campuses.

GROUPING SETS can match either form when you want explicit control. If a report needs the detailed rows, totals by campus, and a grand total, but does not need totals by course level alone, then GROUPING SETS can ask for just that set of grouped levels and skip the rest.

A report like that can be written like this:

SELECT
    campus,
    course_level,
    COUNT(*) AS course_count
FROM class_schedule
GROUP BY GROUPING SETS
(
    (campus, course_level),
    (campus),
    ()
);

This is part of why GROUPING SETS reads well in reporting SQL. The grouped levels are written directly in the query, almost like a compact report outline.

All three forms still depend on regular aggregate logic. You still choose grouped columns. You still choose aggregate expressions such as SUM or COUNT. The change is that the grouped clause can now return multi level summaries in a single result set instead of stopping at a single granularity.

Writing Better Subtotal Queries

Getting multi-level totals into a result set is only part of the work. Reports also need to stay readable after subtotal rows appear. Query-writing choices matter more at that stage. Results can be mathematically right and still feel awkward to read if subtotal rows blur into detail rows, if the grouping form adds levels nobody asked for, or if the SQL relies on syntax that does not travel well between engines. Good subtotal queries keep the output readable while still matching the report layout the query is trying to produce.

Labeling Subtotal Rows

Subtotal rows usually place NULL into grouped columns that are no longer active at that subtotal level. That creates a practical problem right away. Source data can already contain NULL, so subtotal rows with NULL in columns such as region or product_line do not explain themselves. Without a helper function, the output can leave readers unsure which NULL came from missing data and which NULL marks an all-values subtotal.

Readable reports normally start by turning those subtotal markers into text labels people can scan quickly. CASE works well with GROUPING() for that job:

SELECT
    CASE
        WHEN GROUPING(region) = 1 THEN 'All Regions'
        ELSE region
    END AS region_label,
    CASE
        WHEN GROUPING(product_line) = 1 THEN 'All Product Lines'
        ELSE product_line
    END AS product_line_label,
    SUM(revenue) AS total_revenue
FROM sales_fact
GROUP BY ROLLUP(region, product_line)
ORDER BY region_label, product_line_label;

For that, the result reads much better than a report full of unlabeled NULL values. It also keeps actual NULL data separate from subtotal placeholders, which matters when results are exported or passed to charts, spreadsheets, or later review. Row order matters too. Subtotal rows can end up mixed into detail rows in ways that make a report feel scrambled. In engines that support GROUPING_ID(), that function can give you a numeric grouping level that helps keep row order stable:

SELECT
    region,
    product_line,
    SUM(revenue) AS total_revenue,
    GROUPING_ID(region, product_line) AS grouping_level
FROM sales_fact
GROUP BY ROLLUP(region, product_line)
ORDER BY grouping_level, region, product_line;

That ordering tends to place detail rows first, then subtotal rows, then the grand total, which is usually easier to scan than relying on whatever order falls out of the grouped result. Engines without GROUPING_ID() can still produce readable output with GROUPING() inside ORDER BY, though the query gets a little longer.

Label text itself should stay plain. Grand Total, All Regions, All Departments, or similar wording usually works better than labels that force the reader to stop and decode the row. Reports stay easier to follow when subtotal labels match the business language already used in the rest of the output.

Picking the Right Form

Choosing between ROLLUP, CUBE, and GROUPING SETS comes down to the report layout you need. ROLLUP fits best when grouped columns follow a hierarchy. Time-based reporting is a common case because grouped levels usually move from detailed parts up to broader totals. Year, quarter, and month already form that ladder, so ROLLUP reads naturally there.

This kind of query is a good fit for hierarchical totals:

SELECT
    fiscal_year,
    fiscal_quarter,
    fiscal_month,
    SUM(expense_amount) AS total_expense
FROM expense_summary
GROUP BY ROLLUP(fiscal_year, fiscal_quarter, fiscal_month);

In that query, it asks for month-level detail, then quarter totals, then year totals, then the grand total. SQL stays fairly short because the subtotal path is implied by the column order.

CUBE fits a different kind of report. Rather than following a single hierarchy, it returns every grouping combination across the listed columns. That works well when each grouped column can stand on its own as a reporting dimension. Regional totals, channel totals, and product-family totals may all be needed separately, along with the detailed combinations.

That same strength can also produce more output than the report needs. Each added grouped expression raises the number of subtotal combinations, so CUBE is best saved for reports where those combinations are actually wanted. If the report only needs selected subtotal levels, GROUPING SETS is usually the better choice.

Take a query like this:

SELECT
    channel,
    subscription_tier,
    billing_cycle,
    COUNT(*) AS account_count
FROM subscription_accounts
GROUP BY GROUPING SETS
(
    (channel, subscription_tier, billing_cycle),
    (channel, subscription_tier),
    (channel),
    ()
);

That query asks for exactly the subtotal levels needed and nothing extra. That makes GROUPING SETS a strong choice when the report layout is fixed ahead of time and only certain subtotal levels belong in the output.

In short, ROLLUP fits ladder-like totals, CUBE fits every-combination summaries, and GROUPING SETS fits hand-picked subtotal layouts. Picking the form that matches the report keeps the output smaller and keeps the SQL easier to follow later.

Portability Across Major Engines

Support across database engines is close in some places and uneven in others. PostgreSQL, SQL Server, and Oracle support GROUPING SETS, ROLLUP, and CUBE. SQL Server also supports helper functions such as GROUPING() and GROUPING_ID(). Oracle supports both GROUPING() and GROUPING_ID(), and PostgreSQL supports GROUPING(). That gives those three engines fairly similar coverage for subtotal query work, though the helper function details are not identical across them.

MySQL is where portability questions show up faster. Standard MySQL centers more on WITH ROLLUP, so queries written with GROUPING SETS or CUBE do not travel across plain MySQL deployments the same way they do across PostgreSQL, SQL Server, and Oracle. That matters when the same reporting query has to run on more than one backend.

That difference changes how portable SQL should be written. Codebases targeting PostgreSQL, SQL Server, and Oracle can use the full family more freely. When MySQL is part of the target set, the safe subset gets smaller, and subtotal queries may need to rely more on ROLLUP or fall back to UNION ALL for wider compatibility.

Helper functions matter too. SQL Server has both GROUPING() and GROUPING_ID(), which makes labeling and sorting subtotal rows easier. PostgreSQL and Oracle cover the basic grouping marker function, but not every helper is shared the same way across engines. Query text that feels tidy in one engine can need small rewrites in another.

Portability also goes past syntax. Large generated reports can push engine-specific grouping limits or produce more subtotal rows than expected. That is worth keeping in mind when query builders or report layers build grouped SQL automatically instead of by hand.

Direct Alternatives

Grouped extensions are not the only way to write subtotal queries. The most direct fallback is still multiple grouped SELECT statements combined with UNION ALL. That form is longer, but it is familiar SQL and travels well because it only depends on regular grouping and set operations.

This version shows the manual form:

SELECT
    division,
    team_name,
    SUM(ticket_count) AS total_tickets
FROM support_volume
GROUP BY division, team_name

UNION ALL

SELECT
    division,
    NULL AS team_name,
    SUM(ticket_count) AS total_tickets
FROM support_volume
GROUP BY division

UNION ALL

SELECT
    NULL AS division,
    NULL AS team_name,
    SUM(ticket_count) AS total_tickets
FROM support_volume;

In that, the query can produce the same kind of result you would ask from GROUPING SETS. The tradeoff is repetition. Filters, joins, computed expressions, and business rules have to stay aligned across every branch, which leaves more room for drift when somebody edits the query later.

Window functions are another nearby option, but they solve a different problem. Window aggregates attach totals to detail rows without collapsing the result into grouped subtotal rows. That is useful when a report needs every detail row to stay visible while still displaying summary values beside it.

We can see this kind of query in action:

SELECT
    division,
    team_name,
    ticket_id,
    opened_at,
    COUNT(*) OVER (PARTITION BY division) AS division_ticket_total
FROM support_tickets;

That query keeps every ticket row and adds a division total next to it. Useful output, but not the same thing as subtotal rows returned by ROLLUP or GROUPING SETS.

Summary tables and materialized views also belong in the larger discussion. Those options fit recurring reporting workloads where the same subtotal-heavy queries are read repeatedly. That choice is less about subtotal syntax and more about where the summary work should happen, at query time or ahead of time.

Conclusion

GROUPING SETS, ROLLUP, and CUBE all build on normal GROUP BY, but they change what a grouped query can return by letting one statement produce more than one aggregation level. That lets the database generate detail totals, subtotal rows, and a grand total from the same grouped operation instead of forcing separate queries to be stacked by hand. When viewed mechanically, the main difference comes down to how grouping levels are selected, how subtotal rows are marked, and how far the syntax travels across database engines.

1. PostgreSQL GROUP BY and Grouping Sets Documentation

2. SQL Server SELECT GROUP BY Documentation

3. SQL Server GROUPING Documentation

4. SQL Server GROUPING_ID Documentation

5. Oracle Database SELECT Documentation

6. Oracle Database GROUPING Documentation

7. MySQL GROUP BY Modifiers and WITH ROLLUP Documentation

Share Alexander Obregon's Substack

Lots of SQL queries need one related row for each parent row instead of every related row. One customer may need the latest order, one product may need the newest price change, and one account may need the highest payment. Lateral join features fit that job nicely. PostgreSQL gives you LATERAL, SQL Server gives you CROSS APPLY and OUTER APPLY, Oracle supports LATERAL inline views along with APPLY, and current MySQL versions support lateral derived tables. Across those systems, the same idea is in play. The table expression on the right can read values from the current row on the left, then return a small result set for that one row. That makes top-one related row queries much easier to write than piling the whole problem into nested correlated subqueries or large ranking logic right away.

Subscribe now

How Per-Row Lookup Joins Work

Per-row lookup joins solve a very specific SQL problem. One row on the left needs a related lookup before the database moves to the next row. That gives you a natural way to ask for the latest order for one customer, the newest status row for one ticket, or the first matching shipment for one invoice. Instead of building the whole query around a large join and trimming it down later, the lookup can happen as part of the row flow itself. PostgreSQL does that with LATERAL, while SQL Server does it with CROSS APPLY and OUTER APPLY. In both forms, the right side can read columns from the left side row currently being processed.

What LATERAL Changes in PostgreSQL

Regular subqueries in the FROM clause are independent. They act like self-contained table expressions, so they cannot reach across and read columns from neighboring FROM items. PostgreSQL changes that rule with LATERAL. With LATERAL in place, a subquery can reference columns from FROM items that appear before it, and that evaluation happens for each row or row set coming from those earlier items.

That changes how lookup logic can be written. Instead of joining every related row and then working out which one you wanted, you can ask PostgreSQL to take one left row, run a small right-side query tied to that row, and join back only what that small query returns. That is why LATERAL fits per-row lookups so well. The database is not treating the right side like a fixed mini-table. It is treating it as a row-dependent table expression.

Seeing a customer-and-order example can make that flow easy to see:

SELECT
    c.customer_id,
    c.customer_name,
    last_order.order_id,
    last_order.placed_at
FROM customers AS c
LEFT JOIN LATERAL (
    SELECT
        o.order_id,
        o.placed_at
    FROM orders AS o
    WHERE o.customer_id = c.customer_id
    ORDER BY o.placed_at DESC, o.order_id DESC
    LIMIT 1
) AS last_order ON true;

customers is the driving table in that query. For each customer row, the lateral subquery filters orders to that customer, sorts the related rows so the newest one comes first, and keeps only one row with LIMIT 1. LEFT JOIN keeps customers that have no matching orders, so the order columns become null for those rows. The ON true part is common in PostgreSQL lateral join syntax when the row-matching logic already lives inside the subquery.

PostgreSQL also allows LATERAL with table functions in FROM. For row lookup work, though, the common case is still a lateral subquery that filters, sorts, and trims related rows.

Status history is another good fit for the same idea:

SELECT
    t.ticket_id,
    t.subject,
    s.status_code,
    s.changed_at
FROM support_tickets AS t
LEFT JOIN LATERAL (
    SELECT
        h.status_code,
        h.changed_at
    FROM ticket_status_history AS h
    WHERE h.ticket_id = t.ticket_id
    ORDER BY h.changed_at DESC, h.history_id DESC
    LIMIT 1
) AS s ON true;

The flow stays the same. One ticket row goes in, one small history lookup runs, then zero or one chosen history row comes back out.

CROSS APPLY and OUTER APPLY in SQL Server

SQL Server uses APPLY for this family of joins. CROSS APPLY and OUTER APPLY let the right table expression run against each row from the left side. That makes APPLY row-aware in a way that a regular derived table is not. The right side can read values from the current left row and return a rowset tied to that row. If the right side finds one row, that row is attached. If it finds no rows, the final result depends on which APPLY form you picked.

OUTER APPLY keeps the left row and fills the right-side columns with nulls when there is no match. That makes it the SQL Server counterpart to a left-style lateral join.

Take a look at this version:

SELECT
    c.customer_id,
    c.customer_name,
    x.order_id,
    x.placed_at
FROM customers AS c
OUTER APPLY (
    SELECT TOP (1)
        o.order_id,
        o.placed_at
    FROM orders AS o
    WHERE o.customer_id = c.customer_id
    ORDER BY o.placed_at DESC, o.order_id DESC
) AS x;

That query is very close in spirit to the PostgreSQL version, but the SQL Server syntax changes a bit. TOP (1) takes the place of LIMIT 1, and OUTER APPLY keeps customers with no order rows. The sorting logic still decides which related row counts as the chosen one.

This example helps separate CROSS APPLY from OUTER APPLY visually:

SELECT
    p.project_id,
    p.project_name,
    latest_note.note_text
FROM projects AS p
CROSS APPLY (
    SELECT TOP (1)
        n.note_text
    FROM project_notes AS n
    WHERE n.project_id = p.project_id
    ORDER BY n.created_at DESC, n.note_id DESC
) AS latest_note;

That version returns only projects that have at least one note, because CROSS APPLY drops left rows when the right side returns nothing. Changing only the word CROSS to OUTER changes the row-preserving behavior without changing the lookup logic itself.

For a beginner, that is one of the nicest parts of APPLY. The lookup reads like a small query block attached to each left row, so the SQL stays close to the question being answered.

Why This Is Better Than Forcing Everything Into a Scalar Subquery

Scalar subqueries still have a place in SQL, they work well when one left row needs one value and the logic is short. Trouble starts when one chosen related row has several columns you want to return. Scalar subqueries return one value at a time, not a full row from the child table. Lateral joins and APPLY queries return a table expression instead, so several columns can come back from the chosen row without repeating the same lookup over and over.

Let’s imagine we are working with a request for the latest payment per invoice where you need payment_id, paid_at, and amount. With scalar subqueries, one version for each selected value is a common first attempt:

SELECT
    i.invoice_id,
    (
        SELECT p.payment_id
        FROM payments AS p
        WHERE p.invoice_id = i.invoice_id
        ORDER BY p.paid_at DESC, p.payment_id DESC
        LIMIT 1
    ) AS latest_payment_id,
    (
        SELECT p.paid_at
        FROM payments AS p
        WHERE p.invoice_id = i.invoice_id
        ORDER BY p.paid_at DESC, p.payment_id DESC
        LIMIT 1
    ) AS latest_paid_at,
    (
        SELECT p.amount
        FROM payments AS p
        WHERE p.invoice_id = i.invoice_id
        ORDER BY p.paid_at DESC, p.payment_id DESC
        LIMIT 1
    ) AS latest_amount
FROM invoices AS i;

That style repeats the same filtering and sorting logic three times. It also puts more pressure on the reader, who has to scan each subquery to confirm they all pick the same row. One later edit can break that alignment if a tie-breaker changes in one place but not in the others.

A row-producing form reads much better for that case:

SELECT
    i.invoice_id,
    p.payment_id,
    p.paid_at,
    p.amount
FROM invoices AS i
LEFT JOIN LATERAL (
    SELECT
        p.payment_id,
        p.paid_at,
        p.amount
    FROM payments AS p
    WHERE p.invoice_id = i.invoice_id
    ORDER BY p.paid_at DESC, p.payment_id DESC
    LIMIT 1
) AS p ON true;

The row-picking logic appears one time, and all three columns come from the same chosen payment row. That is the big readability gain. SQL Server gets the same benefit with APPLY, and the same idea holds there as well. Scalar subqueries also hide the fact that the query is really asking for a row, not just an isolated value. Per-row lookup joins put that row-level intent in the FROM clause, which is a much better home for row-producing logic.

Inner-Style Behavior

Join behavior matters just as much as the lookup itself. Inner-style behavior means the left row disappears when the right-side lookup returns nothing. PostgreSQL gets that with JOIN LATERAL or CROSS JOIN LATERAL, and SQL Server gets it with CROSS APPLY.

That behavior fits queries where the related row is required. For example, say you want stores that have at least one inventory count entry, and you only care about the latest count for stores that do have one:

SELECT
    s.store_id,
    s.store_name,
    inv.counted_at,
    inv.quantity_on_hand
FROM stores AS s
JOIN LATERAL (
    SELECT
        i.counted_at,
        i.quantity_on_hand
    FROM inventory_counts AS i
    WHERE i.store_id = s.store_id
    ORDER BY i.counted_at DESC, i.count_id DESC
    LIMIT 1
) AS inv ON true;

That query drops stores with no inventory count rows. The join itself is saying that a missing lookup result means there should be no final row.

Left-Style Behavior

Left-style behavior keeps the left row even when the lookup comes back empty. PostgreSQL gets that with LEFT JOIN LATERAL, while SQL Server gets it with OUTER APPLY. That behavior is a better fit for reports where the parent row still matters without a child match. A department list with the latest review row is one example. Departments with no review yet still belong in the result.

This SQL Server version shows that form:

SELECT
    d.department_id,
    d.department_name,
    r.reviewed_at,
    r.score
FROM departments AS d
OUTER APPLY (
    SELECT TOP (1)
        rv.reviewed_at,
        rv.score
    FROM department_reviews AS rv
    WHERE rv.department_id = d.department_id
    ORDER BY rv.reviewed_at DESC, rv.review_id DESC
) AS r;

Departments with no review rows still appear, but reviewed_at and score come back null. That makes the join behavior match the reporting need. The lookup logic on the right did not change much from the earlier examples. What changed was the decision about what to do when no related row exists.

That distinction between inner-style and left-style behavior is one of the first things to settle before writing the query. If the business question says every parent row should still appear, the join form has to reflect that. If the business question says only parents with at least one matching child row belong in the result, the inner-style form is the better fit.

Writing Good Per-Row Lookup Queries Across Systems

Good per-row lookup queries come down to a few practical choices. You need a rule for which related row wins, you need syntax that matches the database product in front of you, and you need supporting indexes that let the database find that row without extra work. The join feature itself is only part of the story. Most mistakes happen earlier, at the point where the query writer has not fully decided what counts as the latest row, the highest row, or the first matching row. After that rule makes sense, the SQL becomes much easier to read and much easier to trust.

Picking the Related Row in a Stable Way

Per-row lookups work best when the winning row is chosen by a rule that stays stable from run to run. Sorting only by a timestamp can leave ties. Sorting only by a score can do the same. If two child rows are tied on the main sort column, the database still needs a consistent way to place one ahead of the other. That is why a second sort column such as an identity value, sequence value, or primary key is usually part of the lookup. Without that extra ordering rule, the query can still run, but the chosen row is not as firmly defined.

A latest subscription event query shows the idea visually:

SELECT
    s.subscription_id,
    s.account_id,
    e.event_type,
    e.recorded_at
FROM subscriptions AS s
LEFT JOIN LATERAL (
    SELECT
        se.event_type,
        se.recorded_at
    FROM subscription_events AS se
    WHERE se.subscription_id = s.subscription_id
    ORDER BY se.recorded_at DESC, se.event_id DESC
    LIMIT 1
) AS e ON true;

recorded_at DESC says newer rows come first. event_id DESC breaks ties when two rows land on the same timestamp. That extra sort column may look minor, but it changes the query from roughly newest row to a fully defined newest row. PostgreSQL processes the lateral subquery per left row, so that ordering rule is applied separately for each subscription.

Some lookups are not about time at all. Highest bid per auction, most expensive invoice line per invoice, or earliest appointment per client all follow the same general structure. What changes is the ordering rule inside the right-side query. SQL Server uses TOP (1) in that spot, while the row ranking logic still depends on the ORDER BY list you provide.

SELECT
    a.auction_id,
    a.title,
    b.bid_id,
    b.bid_amount
FROM auctions AS a
OUTER APPLY (
    SELECT TOP (1)
        ab.bid_id,
        ab.bid_amount
    FROM auction_bids AS ab
    WHERE ab.auction_id = a.auction_id
    ORDER BY ab.bid_amount DESC, ab.bid_id DESC
) AS b;

That query asks for the highest bid for each auction. The primary sort is bid_amount DESC. The secondary sort is bid_id DESC. If two bids tie on amount, the larger bid id wins. Without that second part, tied rows are left to a less explicit rule than most people intend.

Stable row picking also matters when filters appear inside the lookup. A request for the latest paid invoice is different from the latest invoice of any type. That filter belongs inside the right-side query so the database picks from the correct subset first and only then trims to one row. Moving that condition outside can change the result in a way that is easy to miss during a quick read.

PostgreSQL, SQL Server, Oracle, and MySQL Compared

PostgreSQL uses LATERAL for row-dependent subqueries in the FROM clause. Its docs state that a lateral subquery can reference columns from FROM items that appear before it. That is why a row from the left side can feed values into the right-side query during execution. PostgreSQL also allows lateral behavior with table functions in FROM, where the keyword can be optional for functions.

SQL Server expresses the same broad idea with CROSS APPLY and OUTER APPLY. Microsoft’s docs say the right table source is evaluated against each row from the left table source. CROSS APPLY keeps only rows where the right side returns at least one row, while OUTER APPLY keeps the left row and fills the right-side columns with nulls when the lookup returns nothing. That makes APPLY the SQL Server form of a per-row table expression.

Oracle supports both LATERAL inline views and CROSS APPLY and OUTER APPLY. Oracle also has restrictions around how LATERAL can be used, and APPLY syntax should be treated as its own form rather than stacked with LATERAL in the same right-side clause. Put simply, Oracle gives you both syntax families, but they are not meant to be combined in one right-side table expression.

MySQL supports lateral derived tables with the LATERAL keyword in the FROM clause. The MySQL manual states that a lateral derived table can appear only in FROM, and it lists join-side restrictions tied to left and right operands. The manual also notes that JSON_TABLE() is treated as though LATERAL were present, so the keyword is implicit there rather than written explicitly before that function.

These two versions make the syntax contrast easy to see. PostgreSQL looks like this:

SELECT
    p.product_id,
    p.product_name,
    ph.changed_at,
    ph.new_price
FROM products AS p
LEFT JOIN LATERAL (
    SELECT
        h.changed_at,
        h.new_price
    FROM price_history AS h
    WHERE h.product_id = p.product_id
    ORDER BY h.changed_at DESC, h.price_history_id DESC
    LIMIT 1
) AS ph ON true;

SQL Server expresses the same lookup like this:

SELECT
    p.product_id,
    p.product_name,
    ph.changed_at,
    ph.new_price
FROM products AS p
OUTER APPLY (
    SELECT TOP (1)
        h.changed_at,
        h.new_price
    FROM price_history AS h
    WHERE h.product_id = p.product_id
    ORDER BY h.changed_at DESC, h.price_history_id DESC
) AS ph;

The SQL text is different, but the row flow is closely related. One product row is read from the left, a child-row lookup runs for that product, and zero or one selected child row comes back. PostgreSQL and MySQL place that logic under LATERAL. SQL Server places it under APPLY. Oracle accepts both families, subject to its own syntax rules.

Porting these queries from one database to another usually means changing the syntax around the right-side table expression and the row limiting clause. PostgreSQL uses LIMIT 1. SQL Server uses TOP (1). Oracle commonly uses its row limiting clause within the subquery, and MySQL uses LIMIT. The bigger lesson is that the query idea survives the port. Product-specific syntax changes, but the per-row lookup rule stays the same.

Indexing and Performance Habits That Fit This Pattern

Per-row lookup joins tend to work best when the child table has an index that matches both the join filter and the row-picking order. If the query filters child rows by customer_id and then sorts by order_date DESC, order_id DESC, the child-side index usually starts with customer_id and then continues with the sort columns. That gives the database a much better chance to find matching rows quickly and stop after the first qualifying row near the top of the ordered range. Query optimizers differ, but that general index direction lines up well with the lookup itself. Table hints and forced plans are not the starting point here. Good indexing and a well-defined sort rule come first.

Take a current-status lookup for devices:

SELECT
    d.device_id,
    d.serial_number,
    st.status_code,
    st.logged_at
FROM devices AS d
LEFT JOIN LATERAL (
    SELECT
        ds.status_code,
        ds.logged_at
    FROM device_status_log AS ds
    WHERE ds.device_id = d.device_id
    ORDER BY ds.logged_at DESC, ds.status_log_id DESC
    LIMIT 1
) AS st ON true;

An index on the child table that starts with device_id and then includes logged_at and status_log_id fits that query far better than an index that starts only with logged_at. The lookup starts from the parent row, so the index should follow that same direction. The database still makes the final plan choice, but the lookup has a much better starting point when the access path follows the same columns used for filtering and ordering.

Just as important, keep the right-side query focused on picking the row instead of carrying every later report calculation inside it. When the lookup subquery starts doing too much, the row-picking rule gets harder to read. Smaller lookup blocks are easier to verify because the filter and ordering rules stay in one place. If more joins or calculations are needed after the child row is chosen, it is usually better to add them outside the lookup rather than crowding the lookup itself.

One last habit helps keep these queries stable. Decide first what row wins, write that ordering rule in full, make the child-side index support that lookup, and then pick the vendor syntax that matches the database in front of you. That sequence keeps the SQL focused on the result you want instead of drifting into syntax-first writing.

Conclusion

Per-row lookup joins work by letting one row from the left side feed values into a right-side query, which then returns the related row that matches the lookup rule you wrote. That is why LATERAL, CROSS APPLY, and OUTER APPLY fit latest-row, highest-row, and first-match queries so well across different SQL engines. After the row choice is fully defined, the rest comes down to matching the database syntax and giving the child table an index that lines up with the filter and sort order.

1. PostgreSQL Table Expressions and LATERAL

2. SQL Server FROM Clause with APPLY

3. Oracle Database SELECT Statement Reference

4. MySQL Lateral Derived Tables

5. PostgreSQL SELECT Reference

6. SQL Server Query Hints and Table Hints

Share Alexander Obregon's Substack

Normalization structures tables so one fact lives in one place, which helps inserts, updates, and deletes avoid conflicting copies of the same data. It starts with spotting repeated groups and relationships that are hiding inside wide rows, then splitting that data into tables that match how the rows relate to each other, so changes stay consistent without side effects.

Subscribe now

Why Normalization Exists

In short, normalization keeps facts in one place so the database does not store conflicting copies of the same information. Tables start drifting when one row tries to represent more than one thing at the same time, such as an order, the items on that order, and customer details. That drift usually appears in two ways repeated groups that grow sideways into extra columns, and anomalies where routine edits create mismatched data.

Repeated Groups Show Up as Extra Columns or Mixed Lists

Wide rows can hide repeated groups when the same set of fields repeats side by side, like item1_*, item2_*, and so on. Another form shows up when a table stores a list inside one column, like SKUs joined by commas. Both styles hide the same relationship. One order can contain multiple items, yet the table is laid out like an order can only hold a fixed number of items.

An early draft order table looks like this because it’s normal to keep everything together in one record:

CREATE TABLE order_sheet (
  order_id        BIGINT PRIMARY KEY,
  order_date      DATE NOT NULL,
  customer_id     BIGINT NOT NULL,
  customer_name   VARCHAR(255) NOT NULL,
  customer_phone  VARCHAR(32),
  item1_sku       VARCHAR(50),
  item1_name      VARCHAR(255),
  item1_price     NUMERIC(10,2),
  item1_qty       INTEGER,
  item2_sku       VARCHAR(50),
  item2_name      VARCHAR(255),
  item2_price     NUMERIC(10,2),
  item2_qty       INTEGER
);

Pressure shows up quickly. First, a hard ceiling appears on how much one order can hold, so extra items push you toward adding item3_* columns or splitting one order into multiple rows, both of which blur what order_id is supposed to mean. Second, the layout stops matching the questions you want to ask. Searching for orders that contain a product turns into a scan across multiple columns, and the query gets longer every time more item columns are added:

SELECT order_id, order_date
FROM order_sheet
WHERE item1_sku = 'SKU-1007'
   OR item2_sku = 'SKU-1007';

Total calculations run into similar trouble. Once line counts vary, totals turn into repeated expressions with lots of null handling, which is a sign the table is carrying a repeating structure that wants rows instead of columns.

Lists inside one column cause the same problems in a different disguise. One column that holds values like SKU-1007,SKU-2040 cannot be validated with a normal foreign key, cannot be joined safely to a product table, and turns searches into string work. That pushes business rules into ad hoc parsing logic, and different queries start parsing the same column in different ways.

Anomalies Signal Structural Trouble

Problems show up fast when routine changes force edits across more rows than expected, when recording one fact requires inventing unrelated facts, or when deleting one row removes data you did not mean to remove.

Update anomalies appear when one fact is stored in multiple places. In the order_sheet table, customer data is copied into each order row. If a customer changes their phone number, the edit needs to touch every row that repeated that value. Missing one row leaves the database claiming two phone numbers for the same customer.

Two updates that do not hit the same set of rows can create a mismatch fast:

UPDATE order_sheet
SET customer_phone = '715-555-0199'
WHERE customer_id = 501;

UPDATE order_sheet
SET customer_phone = '715-555-0188'
WHERE customer_id = 501
  AND order_date >= DATE '2026-02-01';

After those statements, customer_id 501 has two phone numbers stored in the same table. Nothing in the table structure prevents it, because the layout treats customer details as if they were order details.

Insert anomalies show up when the table layout blocks you from recording something that should stand on its own. If product details only exist inside item1_name, item1_price, and similar columns, adding a new product to a catalog becomes impossible without also creating an order row, because the table has no separate home for product facts. People work around that by inserting placeholder orders or placeholder customers, which pollutes data quality.

Delete anomalies show up when removing one row also removes facts that were not part of the deletion decision. If the only place a product name exists is inside old order rows, deleting old orders can erase the last record of what that product was called. That is a side effect of storing multiple concepts in one row, where each concept should have its own table.

First Normal Form Through Third Normal Form With One Running Example

This part takes the order story and reshapes it step by step so the tables match the relationships in the data. The same order can contain multiple line items, product details belong to a product catalog, and customer details belong to the customer record. Normal forms give names to those separations so you can reason about them and spot trouble early.

First Normal Form Make Rows Atomic

Row structure comes first. Values stay atomic, meaning one value per column, and repeating groups stop living as extra column sets like item1_* and item2_*. Instead of widening the table each time an order has more items, line items become rows in their own table, tied back to the order. The table split below keeps order level facts in orders and item level facts in order_item. line_no gives each item a stable identity inside its order, which matters when the same SKU appears twice for separate lines or when line order matters on invoices.

CREATE TABLE orders (
  order_id     BIGINT PRIMARY KEY,
  order_date   DATE NOT NULL,
  customer_id  BIGINT NOT NULL
);

CREATE TABLE order_item (
  order_id    BIGINT NOT NULL,
  line_no     INTEGER NOT NULL,
  sku         VARCHAR(50) NOT NULL,
  qty         INTEGER NOT NULL,
  unit_price  NUMERIC(10,2) NOT NULL,
  PRIMARY KEY (order_id, line_no),
  FOREIGN KEY (order_id) REFERENCES orders(order_id),
  CHECK (qty > 0),
  CHECK (unit_price >= 0)
);

After that split, any order can carry as many items as needed without altering the schema. Searches also get easier because the SKU lives in one column across all line rows, rather than spread across item1_sku, item2_sku, and future columns.

Data entry also starts to be a bit more natural. Insert the order once, then insert one row per line item:

INSERT INTO orders (order_id, order_date, customer_id)
VALUES (90001, DATE '2026-02-10', 501);

INSERT INTO order_item (order_id, line_no, sku, qty, unit_price)
VALUES
  (90001, 1, 'SKU-1007', 2, 19.99),
  (90001, 2, 'SKU-2040', 1, 8.50),
  (90001, 3, 'SKU-1007', 1, 17.99);

That last line item repeats the same SKU on a separate line with a different unit_price. That can happen with discounts, price changes, or partial returns. A wide table with one SKU slot per column set struggles with that case because the repeated group is trying to represent a list, not a fixed set of columns.

Second Normal Form Remove Partial Dependencies From Composite Identifiers

Composite identifiers deserve extra attention. order_item identifies a row by (order_id, line_no), which means any non identifying column in that table should depend on the full pair, not only part of it. Quantities and invoice prices belong to that line within that order, so they fit naturally.

Order level values like order_date and customer_id depend on order_id alone, so they stay in orders instead of being repeated on every order_item row identified by (order_id, line_no). That keeps one edit from turning into many edits, and it prevents line rows from disagreeing about the same order.

After that, you can split catalog facts into product so the name and current list_price live with the sku, while order_item keeps the invoice unit_price that reflects what was charged on that order.

CREATE TABLE product (
  sku         VARCHAR(50) PRIMARY KEY,
  name        VARCHAR(255) NOT NULL,
  list_price  NUMERIC(10,2) NOT NULL,
  CHECK (list_price >= 0)
);

ALTER TABLE order_item
  ADD CONSTRAINT order_item_product_fk
  FOREIGN KEY (sku) REFERENCES product(sku);

order_item can keep sku, qty, and the invoice unit_price, while the current catalog name comes from joining to product when needed. Put a foreign key on order_item.sku so the database can validate that every line item references a real catalog row. That division also supports a common business rule. list_price can change over time, while unit_price on the order line reflects what was charged on that date.

Third Normal Form Remove Transitive Dependencies

Transitive dependencies show up when a non identifying column depends on another non identifying column. Customer attributes in an order table are the classic case. When orders stores customer_id, any customer name or phone stored in orders depends on customer_id, not on order_id. That creates duplicated customer facts across order rows.

Customer attributes belong in a customer table, with orders referencing it:

CREATE TABLE customer (
  customer_id  BIGINT PRIMARY KEY,
  name         VARCHAR(255) NOT NULL,
  phone        VARCHAR(32)
);

ALTER TABLE orders
  ADD CONSTRAINT orders_customer_fk
  FOREIGN KEY (customer_id) REFERENCES customer(customer_id);

With that structure, changing a phone number becomes one update in one place, and all orders that reference the customer still point at the current contact data. Joins in reports pull the related rows back into one result set so you can view an order with its customer details in a single query:

SELECT
  o.order_id,
  o.order_date,
  c.name AS customer_name,
  c.phone AS customer_phone
FROM orders o
JOIN customer c ON c.customer_id = o.customer_id
WHERE o.order_date >= DATE '2026-01-01'
ORDER BY o.order_date, o.order_id;

Some data looks like customer data but is actually order data. Shipping address is a good example. Customers can have multiple addresses, orders can ship to a one time destination, and historical invoices usually need the exact address that was used at the time. Normalization does not force one choice there. It pushes you to decide what the column represents. If the value represents the customer profile, store it with the customer. If the value represents what happened on the order, store it with the order.

Spotting Junction Table Needs

Junction tables come up when two entity tables relate in a many to many way. Promotions and orders fit that structure well, because an order can have multiple promotions applied, and a promotion can apply to multiple orders. Repeated columns like promo_code1 and promo_code2 run into the same repeated group problem as item columns, and a comma separated list runs into the same validation and query problems as any list stored in a single column.

With a junction table, the relationship is stored as rows instead of being packed into columns or text:

CREATE TABLE promotion (
  promo_code  VARCHAR(50) PRIMARY KEY,
  description VARCHAR(255) NOT NULL
);

CREATE TABLE order_promotion (
  order_id    BIGINT NOT NULL,
  promo_code  VARCHAR(50) NOT NULL,
  PRIMARY KEY (order_id, promo_code),
  FOREIGN KEY (order_id) REFERENCES orders(order_id),
  FOREIGN KEY (promo_code) REFERENCES promotion(promo_code)
);

That structure supports questions that are awkward in a wide column layout, such as finding all orders that used a given promotion or counting promotion usage over a date range. It also supports validation. Foreign keys can confirm that every promo_code recorded on an order exists in the promotion table, and the composite primary identifier prevents duplicate application of the same promotion to the same order unless you intentionally model repeated application as its own concept.

Data entry follows the same rhythm as line items. Insert promotions into promotion, then record pairings in order_promotion as they happen on orders:

INSERT INTO promotion (promo_code, description)
VALUES
  ('WELCOME10', 'New customer discount'),
  ('FREESHIP', 'Free shipping promotion');

INSERT INTO order_promotion (order_id, promo_code)
VALUES
  (90001, 'WELCOME10'),
  (90001, 'FREESHIP');

After the relationship is stored as rows, the database can answer relationship questions through joins and filtering, rather than through column scanning or string parsing.

Conclusion

Normalization works by turning hidden repetition into explicit relationships that the database can store and join reliably. First normal form moves repeating groups out of wide rows so line items become their own rows tied back to an order. Second normal form keeps order level facts off line item rows when the line item table is identified by a multi column identifier, so values like order_date and customer_id stay on the single order row in orders instead of being repeated across lines. Third normal form separates customer facts from order facts, so customer attributes change in one place while orders keep the history of order facts. If you need historical customer name or phone as it was at purchase time, store a snapshot on the order or in a separate history table. When the relationship is many to many, the junction table keeps each pairing in its own row, so constraints can validate the data and queries stay join based instead of scanning extra columns or splitting strings.

1. PostgreSQL Documentation on CREATE TABLE

2. PostgreSQL Documentation on Constraints

3. PostgreSQL Documentation on Foreign Keys

4. SQL Server Documentation on CREATE TABLE

5. MySQL Documentation on InnoDB Foreign Keys

6. SQLite Documentation on Foreign Key Support

Share Alexander Obregon's Substack

In general, SQL runs best when you tell the database which rows you want and what result you want back, then let the optimizer choose a plan to read and transform that set. That set based style is the default for good reasons, but stored routines still sometimes need row by row work for stateful steps, external calls, or ordered processing that you cannot express as one statement.

Subscribe now

Set First Thinking Before Reaching for a Cursor

Cursor work starts making sense only after you have already done the set work. Think of it as two phases that stay separate. Phase one is a query that narrows the work down to the rows that matter and returns only the columns you need. Phase two is the procedural step that runs per row, usually because it has state, side effects, or per row error handling that a single statement cannot express well.

Keeping that split upfront keeps the cursor small and keeps the database doing what it is built to do, which is filter, join, and project sets of rows efficiently. It also helps you test the input set on its own before any procedural logic touches it.

What Set Based SQL Is Built to Do

Set based SQL focuses on describing results, not dictating step order. You express a relationship between tables, filters, and projections, then the optimizer decides how to reach that result. That freedom is what allows join reordering, predicate pushdown, index selection, streaming operators, and parallel execution where the engine supports it. One practical way to see the difference is to compare a single statement that updates a set of rows against a loop that updates one row at a time.

This is a typical set based update in SQL Server. It touches every matching row in one statement, and the engine chooses a plan that matches the table sizes and indexes:

UPDATE a
SET a.StatusId = w.NewStatus,
    a.UpdatedAt = SYSUTCDATETIME()
FROM dbo.Account AS a
JOIN dbo.AccountStatusWork AS w
  ON w.AccountId = a.AccountId
WHERE w.ProcessedAt IS NULL;

That statement does not lock you into one join method or one access path. It still has a defined result, but the path to that result is chosen by the optimizer based on statistics, indexes, and the current environment. That is the main reason set based SQL tends to outperform procedural loops for pure data movement.

Ordering fits the same idea, a query only has an ordered output when you ask for it with ORDER BY. The database can read data in whatever internal order is efficient, then sort or otherwise order the final output stream when required.

SELECT TOP (50) w.AccountId, w.NewStatus
FROM dbo.AccountStatusWork AS w
WHERE w.ProcessedAt IS NULL
ORDER BY w.AccountId;

That query gives you predictable ordering in the result, while still letting the engine pick the best way to locate the rows.

When Row by Row Work Still Makes Sense

Row by row work earns its place when each row triggers a procedural action, when the next step depends on state that changes as you go, or when you need per row logging and error capture. That is where cursor style processing fits, because it gives you a controlled sequence over a query result set.

Practical work goes better when the per row stage is fed by a tight set producing query. Work tables are common for this, you populate them in a set based statement, then process the queued rows one at a time.

INSERT INTO dbo.AccountStatusWork (AccountId, NewStatus, CreatedAt)
SELECT a.AccountId,
       CASE WHEN a.IsLocked = 1 THEN 5 ELSE 2 END,
       SYSUTCDATETIME()
FROM dbo.Account AS a
WHERE a.UpdatedAt < DATEADD(day, -7, SYSUTCDATETIME());

After the input set is defined, the per row loop can focus on the procedural action and progress tracking rather than searching for the next row.

DECLARE @AccountId int;
DECLARE @NewStatus int;

DECLARE work_cursor CURSOR LOCAL FAST_FORWARD FOR
SELECT w.AccountId, w.NewStatus
FROM dbo.AccountStatusWork AS w
WHERE w.ProcessedAt IS NULL
ORDER BY w.AccountId;

OPEN work_cursor;

FETCH NEXT FROM work_cursor INTO @AccountId, @NewStatus;

WHILE @@FETCH_STATUS = 0
BEGIN
    UPDATE dbo.Account
    SET StatusId = @NewStatus,
        UpdatedAt = SYSUTCDATETIME()
    WHERE AccountId = @AccountId;

    UPDATE dbo.AccountStatusWork
    SET ProcessedAt = SYSUTCDATETIME()
    WHERE AccountId = @AccountId;

    FETCH NEXT FROM work_cursor INTO @AccountId, @NewStatus;
END

That is still a cursor loop, but it is a cursor loop over a pre filtered set, which keeps the procedural part focused on the row by row requirement rather than doing set work repeatedly inside the loop.

Cursor Lifecycle From Declare to Deallocate

Row by row work through a cursor follows a lifecycle with defined stages, and each stage has a specific meaning. Declaration ties a name to a query and options. Opening activates the cursor so rows can be read. Fetching steps through rows in a controlled order. Closing releases the active result set. Deallocation releases cursor resources so the cursor name is no longer valid for further operations.

Declaration Then Open

At the DECLARE step, you define what the cursor will read and how it will behave. In SQL Server, DECLARE CURSOR binds the cursor name to a SELECT statement and cursor options. Options matter because they affect behavior and overhead. For row by row processing that only needs forward movement and no updates through the cursor, FAST_FORWARD is a common choice because it implies a FORWARD_ONLY, READ_ONLY cursor.

Code tends to read well when the cursor name reflects the row source and the variables match the projected columns:

DECLARE @WorkId int;
DECLARE @AccountId int;

DECLARE work_cursor CURSOR LOCAL FAST_FORWARD FOR
SELECT w.WorkId, w.AccountId
FROM dbo.AccountWorkQueue AS w
WHERE w.ProcessedAt IS NULL
ORDER BY w.WorkId;

OPEN work_cursor;

OPEN activates the cursor and positions it before the first row in the result set. Data does not come back at OPEN time. Rows arrive when you FETCH.

PostgreSQL has a similar separation. DECLARE cursor_name CURSOR FOR defines the cursor and its query, and cursor lifetime is tied to a transaction unless WITH HOLD is specified. That transaction tie is the reason you commonly start a transaction block before declaring the cursor, then keep the cursor work inside that same transaction scope.

BEGIN;

DECLARE work_cursor NO SCROLL CURSOR FOR
SELECT work_id, account_id
FROM account_work_queue
WHERE processed_at IS NULL
ORDER BY work_id;

Cursor options in PostgreSQL also matter. Forward only work fits well with NO SCROLL. If you need backward movement, PostgreSQL expects the cursor to be declared SCROLL. Relying on backward fetch behavior without SCROLL is not a good idea, and staying explicit keeps behavior consistent.

MySQL has a different constraint that shapes the lifecycle. MySQL cursor support is tied to stored programs, so cursor declaration and fetch loops live inside stored procedures or functions rather than ad hoc SQL.

Fetch Loop Mechanics

Row retrieval happens during FETCH. Each fetch copies the current row into variables and moves the cursor position forward based on the direction you requested. In SQL Server, FETCH NEXT is the common forward movement option. The first FETCH NEXT after OPEN returns the first row. Loop control usually checks @@FETCH_STATUS after each fetch, and a status of 0 means the fetch succeeded.

In practice, the loop pulls one row into variables, runs the per row work, then pulls the next row:

FETCH NEXT FROM work_cursor INTO @WorkId, @AccountId;

WHILE @@FETCH_STATUS = 0
BEGIN
    UPDATE dbo.AccountWorkQueue
    SET PickedAt = SYSUTCDATETIME()
    WHERE WorkId = @WorkId;

    EXEC dbo.ProcessAccountWork @AccountId = @AccountId;

    UPDATE dbo.AccountWorkQueue
    SET ProcessedAt = SYSUTCDATETIME()
    WHERE WorkId = @WorkId;

    FETCH NEXT FROM work_cursor INTO @WorkId, @AccountId;
END

PostgreSQL fetch behavior is expressed through FETCH against the cursor name. In PL pgSQL, routines frequently rely on FOUND or NOT FOUND after a FETCH to decide when to exit a loop. PostgreSQL also supports MOVE, which is like FETCH in terms of cursor movement but does not return a row, so it is useful when you need to reposition the cursor without pulling data into variables.

Something with PostgreSQL to keep in mind is that PL pgSQL cursor docs mark bare count expressions without an explicit direction keyword as deprecated, so stick with forms like FETCH NEXT or FETCH FORWARD n.

Close Deallocate Scope

Closing and deallocation are separate steps, and treating them as separate steps keeps intent easy to follow.

SQL Server CLOSE releases the current result set associated with the cursor. The cursor still exists after CLOSE, and it can be opened again if needed. DEALLOCATE releases the cursor resources and removes the cursor definition, so reopening is no longer possible without a new DECLARE CURSOR.

CLOSE work_cursor;
DEALLOCATE work_cursor;

Scope rules matter in SQL Server. LOCAL limits the cursor to the current scope. GLOBAL makes it visible beyond the current scope. Picking the smallest scope that fits reduces surprises when code is refactored into nested procedures.

PostgreSQL has a different scope anchor. Cursors typically live within the current transaction. CLOSE frees resources associated with an open cursor, and after closing, further operations on that cursor are not allowed. If the cursor is not declared WITH HOLD, committing the transaction ends the cursor lifetime as well.

Error Handling Batch Work Without Bottlenecks

Row by row routines fail in a different way than set based statements. Failures can happen on one item while the next item is fine, and that usually calls for per row logging and per row decisions about retry, skip, or stop.

SQL Server TRY and CATCH blocks make per row error capture easy. One common pattern logs the error message alongside the work item id, then continues to the next fetch. Progress tracking stays separate from the business work so you can see what was picked, what finished, and what failed.

One way to keep the flow readable is to update the work row in both branches so the queue table tells the full story:

DECLARE @WorkId int;
DECLARE @AccountId int;

FETCH NEXT FROM work_cursor INTO @WorkId, @AccountId;

WHILE @@FETCH_STATUS = 0
BEGIN
    BEGIN TRY
        EXEC dbo.ProcessAccountWork @AccountId = @AccountId;

        UPDATE dbo.AccountWorkQueue
        SET ProcessedAt = SYSUTCDATETIME(),
            ErrorText = NULL
        WHERE WorkId = @WorkId;
    END TRY
    BEGIN CATCH
        UPDATE dbo.AccountWorkQueue
        SET ProcessedAt = SYSUTCDATETIME(),
            ErrorText = ERROR_MESSAGE()
        WHERE WorkId = @WorkId;
    END CATCH;

    FETCH NEXT FROM work_cursor INTO @WorkId, @AccountId;
END

Batching keeps cursor work from becoming a long running transaction that holds locks too long. Practical batch flow marks a limited number of rows as picked, processes only those rows, then commits and repeats. That also helps when a routine is interrupted, because the work table can track picked time, processed time, and error text.

PostgreSQL batch patterns commonly rely on a work table plus row locking so multiple workers do not pick the same items. FOR UPDATE SKIP LOCKED is frequently used to claim rows without waiting on locks, then the routine processes the claimed set. Cursor based processing still fits that style, but transaction boundaries matter more because the cursor and locks live inside the transaction.

Batching is also where cursor options and cursor scope connect back to resource usage. Smaller batches limit the active cursor window, reduce lock duration, and reduce the chance that one slow row blocks progress for everything behind it.

Conclusion

Cursors add a procedural layer on top of SQL’s set based engine, so the mechanics matter. Start with a query that defines the input rows, then walk through the lifecycle in order DECLARE to bind the cursor to that query, OPEN to activate it, FETCH to pull rows into variables while moving the cursor forward, CLOSE to release the active result set, and DEALLOCATE to release cursor resources. Keeping the set producing query tight, keeping the per row work focused, and batching when needed keeps the cursor stage predictable while the database still does the heavy lifting where it performs best.

1. SQL Server DECLARE CURSOR Transact SQL

2. SQL Server FETCH Transact SQL

3. SQL Server CLOSE Cursor Transact SQL

4. SQL Server DEALLOCATE Cursor Transact SQL

5. PostgreSQL DECLARE

6. PostgreSQL FETCH

7. PostgreSQL CLOSE

8. PL pgSQL Cursors

9. MySQL Cursors

Share Alexander Obregon's Substack

Sending SQL as a template and the user values as separate inputs lets the database treat the SQL as code while keeping the values as data. That division changes the whole trip from client to server because the driver sends structured parameter values instead of pasting text into a string, then the database parses and compiles the statement without letting those values alter the grammar. SQL injection attempts fall apart for the same reason, and plan caching behavior shifts too, which matters in engines like SQL Server and PostgreSQL.

Subscribe now

Parameter Binding Mechanics From Client Code

Most application database calls follow the same basic plan. You write one SQL statement that stays stable over time, then you pass values separately at runtime. The database receives both pieces in a structured form and connects them at execution time, which changes what the server parses, what the driver sends, and what you can safely allow user input to do.

Placeholders and Typed Values

Different databases and drivers spell placeholders differently, but the intent is consistent. SQL Server commonly uses named parameters like @customerId inside the SQL text. PostgreSQL’s server protocol uses positional parameters like $1, $2. Client libraries can expose different placeholder spellings in application code, like psycopg’s %s or node-postgres $1, but the binding is still positional and the values still travel separately from the SQL text.

Typed values matter more than people expect at first. The driver is not just shipping a string of characters. It’s sending a value along with a type, or at least a strong hint about the type. That type affects how the database parses the statement, how it checks conversions, and how it compares values inside operators and predicates.

With SQL Server client code, you’ll usually see a parameter created with a declared SqlDbType. It changes what goes over the connection and how SQL Server reads it.

Read more

Sooner or later, every database ends up answering the same set of questions after data changes. Which row changed, who made the change, and what the row looked like right before it happened. Triggers give you a place inside the database to capture that record at write time, right when an insert, update, or delete runs, so the audit data stays tied to the same transaction as the change. Trigger based auditing then comes down to writing the old values, new values, a timestamp, and whatever user identity the engine can expose into an audit table you can query later.

Subscribe now

How Trigger Auditing Works

Trigger auditing runs alongside the write path inside the database, so the work happens at the same moment the change happens. When a row change statement runs, the engine builds the before and after row images it already needs for the operation, and trigger code can read those images and write a matching record into a separate table.

Trigger Timing and Row Images

Timing decides when trigger code runs relative to the row change. Many engines support BEFORE and AFTER timing, and the choice affects what trigger code can safely assume about the final state. AFTER timing is a common fit for auditing because the engine has already accepted the change when the trigger body runs, so the audit insert records what the table ended up with. BEFORE timing still has a place, but it can invite extra logic that starts to look like business rules rather than audit capture. Row images are the other half of the picture. Auditing needs access to values from the row before the change and values after the change, and each database surfaces that data in its own way.

PostgreSQL row level triggers expose special record values named OLD and NEW, and the trigger function can also read variables like TG_OP and TG_TABLE_NAME to learn what fired it.

MySQL triggers also expose OLD and NEW inside the trigger body, where OLD.col_name refers to the prior row value for updates and deletes, and NEW.col_name refers to the inserted or updated row value.

SQL Server exposes row images through two logical tables inside the trigger, inserted and deleted. An INSERT statement places rows in inserted, a DELETE statement places rows in deleted, and an UPDATE statement places old rows in deleted and new rows in inserted.

Let’s see how trigger timing changes what the database hands your trigger during an update, and how OLD and NEW show up inside a PostgreSQL trigger function:

Read more

Almost all database write work takes more than one statement. Features usually touch several rows, and sometimes more than one table, so the changes need to land as one unit or not land at all. Transactions handle that. You wrap a group of writes, then either COMMIT to make them permanent or ROLLBACK to undo the row changes, though some databases still consume generated id or sequence values even if you roll back. That’s what keeps a money transfer from ending with money removed from one account but never added to the other after a mid way failure.

Subscribe now

Transaction Basics

Multi step writes need a boundary that treats several statements as one unit. That boundary is the transaction. Inside it, your session can stage changes, read them back, and decide to make them permanent or throw them away. Getting comfortable with that mental model makes the rest of the topic feel far less slippery.

What a Transaction Really Means

Think of a transaction as a temporary workspace tied to your current database session. Statements run the same way they normally do, but their effects stay private to your session until you decide what to do at the end. You can run several INSERT, UPDATE, and DELETE statements, then finish with COMMIT to publish the result. If something goes wrong, ROLLBACK throws away everything done after the transaction began.

One practical detail can help a lot early on is that your session can see its own uncommitted changes, and other sessions usually cannot unless they are running a mode that allows dirty reads. So after an UPDATE, a SELECT in the same transaction will usually read back the new value, even though nobody else can see it yet. Transactions also provide an all or nothing result. If step three fails, steps one and two do not get left behind when you roll back.

Let’s say we are working with two sessions. Session 1 starts a transaction and updates a row. Session 2 reads at the same time. Session 2 keeps seeing the older committed value until Session 1 commits.

Run this in Session 1:

Read more

Upsert work comes up when changes keep arriving and you need one table to stay current. That could be a nightly file load, a CDC feed, or a staging table you refresh each run. SQL’s MERGE statement does that job well because you write the match rule one time, then outline what to do for rows that match and rows that don’t.

Subscribe now

How the T SQL MERGE Statement Syncs a Target

MERGE gives you one place to express how rows line up between a source set and a target table, then what to do with matches and non matches. It reads like a data sync plan, but it still runs as a join driven DML operation, so the match rule and the source set matter a lot.

Join Shape for Matching Rows

Thinking of MERGE as a join helps. SQL Server builds a row set out of the target and the source by applying the ON predicate, then it decides which action to take for each source row based on match status. The ON predicate is the definition of what counts as the same business row, so it should line up with a PRIMARY KEY or UNIQUE constraint on the target whenever that is possible. When the match rule is too broad, one target row can end up paired with more than one source row. When the source contains duplicates for the same identifier, the source can try to update or insert the same logical row more than once in a single statement, and MERGE can fail or behave in ways that are hard to reason about during a load review.

Before writing MERGE, it helps to sanity check the source set in the same way you would check input to a bulk load. Look for duplicates on the identifier you plan to match on, then decide how to collapse them. In staging pipelines, the most common rule is to keep the newest row per identifier based on a timestamp or a sequence value. That gives the ON predicate a stable match on one row per identity and keeps the update path deterministic.

One common collapse method is a ROW_NUMBER filter that keeps the newest row per business identifier.

Read more

Database servers still rely on stored procedures to group SQL statements, keep business rules close to the data, and present stable entry points to application code in the database catalog that accept parameters, run one or more statements, and return result sets or single value outputs. This arrangement keeps repeated queries in one place, supports performance for large batches, and lets database developers treat procedures as a data API between application code and the physical tables.

Subscribe now

What Stored Procedures Do In Databases

Stored procedures give a database its own library of reusable queries and data operations. They live next to the tables and indexes, have names, accept parameters, and can return rows or single values. Instead of sending a long SQL string from the application every time, a client can call a stored procedure by name and hand in the values that change, while the database keeps the actual logic in one place. That setup keeps query text consistent, helps with plan reuse, and gives database staff a stable place to express business rules that must live close to the data.

Relational engines treat stored procedures as objects in the catalog. SQL Server places them in schemas such as dbo, PostgreSQL tracks them in its system catalogs, MySQL stores them in its data dictionary, and Oracle manages them in its data dictionary views. The general flow is similar across engines. A procedure is created with a DDL statement, compiled into an internal representation, and then called many times with different arguments. The caller does not need to send the full text again, only the procedure name and parameter values.

Stored procedures can do far more than a single SELECT. They can chain multiple statements, such as an insert followed by an update, and they can branch on conditions with IF or CASE. Many teams use them for write heavy operations that must run in a controlled sequence, for read heavy logic that performs several related queries, and for routines that sit at the boundary between an application and its core data.

Stored Procedure Definition

Definitions start by giving the procedure a name and a parameter list, then a block of statements that run when the name is called. Many real databases use schemas to group procedures that relate to a given area of the application, such as billing or inventory, which helps keep long catalogs organized.

Here is a stored procedure in SQL Server that returns recent orders for a single customer:

This definition registers dbo.GetRecentOrders as a stored procedure in the database. The two parameters describe the values the caller must supply. SET NOCOUNT ON asks SQL Server not to send row count messages after each statement, which keeps result traffic focused on the data. The SELECT filters rows based on those parameters and returns the columns that callers care about. When client code needs recent orders, it can call dbo.GetRecentOrders instead of embedding the full SELECT each time.

Executions of this procedure reuse the same compiled plan in SQL Server unless something important changes, such as table statistics or certain schema details. Plan reuse helps with performance because the server does not reparse and replan the statement text on every call. Parameter values still influence row estimates and index paths, but the overall skeleton of the query remains stable across calls.

PostgreSQL supports functions that return result sets, which many teams use the same way they use stored procedures in SQL Server:

And for this function, it lives in the database catalog and can be called from SQL with SELECT * FROM get_recent_orders(7, 30);. The RETURNS TABLE clause tells PostgreSQL what columns to expect, which supports result type checking for callers. For LANGUAGE sql functions, the planner can inline the function into the calling query when it is safe to do so, so planning happens as part of the caller. For plpgsql, SQL statements written directly in the function body are planned and cached for reuse, and they get replanned when invalidated. Dynamic SQL run with EXECUTE is planned each time it runs.

MySQL uses a slightly different syntax, but the idea is the same. Client tools usually change the statement delimiter so the body can contain semicolons without ending the entire CREATE PROCEDURE command:

This definition gives MySQL a procedure named GetCustomerBalance that accepts one parameter and returns a single row with a balance column. The database stores this definition and makes it callable through CALL GetCustomerBalance(7);. That call reads more like a function call than a raw query, which suits code that wants a named operation rather than a text string.

Across different engines, stored procedure definitions usually support control flow as well. It is common to see IF branches that choose among several queries, loops that walk through cursor results when set based statements are not practical, and RETURN or RAISE statements that exit with status codes. Those features make procedures closer to small programs that live inside the database, rather than single ad hoc queries.

Parameters With Output Values

Business logic rarely stops at a fixed query. Procedures tend to accept parameters to filter data, control date ranges, or select among modes of operation. Input parameters act as placeholders in the stored code and receive their values from the caller. That arrangement lets the database reuse one definition while still handling many different cases.

Output parameters provide a second channel for returning values beside result sets. A procedure can stream rows in a normal result set and still set one or more output parameters for totals, counts, or status codes. Callers then read those output values the same way they read any other parameter, without parsing a special result row.

Here is a SQL Server procedure that accepts a customer id as input and exposes the total order amount as an output value:

The first SELECT returns detailed rows to the caller, one per order. The second query computes the total and assigns it into @TotalAmount, which then leaves the procedure as an output parameter. Client code receives two pieces of information in one call, both the list and the aggregate number. This pattern avoids an extra round trip just to compute the total.

Return codes often sit next to output parameters in SQL Server. Many teams add an INT return value that represents success or specific error conditions, while still raising errors for severe problems. That return value can be used in calling T-SQL or in client code to make decisions about retries or alternate flows.

PostgreSQL procedures and functions can also use input and output parameters. Functions that use OUT parameters return those values as columns in the result. This example computes a count and a total amount for a customer in a single call:

Calling SELECT * FROM get_customer_order_stats(7); returns a single row with order_count and total_amount. Input and output values tie directly to the parameter list, which keeps the function signature self documenting. Client code that calls this function does not need to know the internal query, only the name and the output column names.

Default parameter values give procedures extra flexibility. Databases such as SQL Server and PostgreSQL support defaults, which let callers omit certain arguments when they want common behavior. Procedures that usually look back thirty days can define @DaysBack INT = 30 or p_days_back INT DEFAULT 30. Callers that want the default window do not have to pass a value, while callers with special needs can pass a different number.

Write focused stored procedures rely heavily on parameters and outputs. In one routine that creates an order, the procedure may accept customer data, product ids, quantities, and discount codes as inputs, then hand back a new order id as an output parameter or a single row result. Journal posting procedures can return a transcript id through an output parameter so later processes can refer back to it. These patterns keep the work anchored inside the database while still giving applications clear ways to send in values and read back results.

Working With Stored Procedures In Applications

Client code that talks to a database does not need to send raw SQL text for every operation. Stored procedures give that code a named entry point that represents a unit of work inside the database, such as loading a summary view, posting a payment, or archiving a set of records. The application passes parameters, the procedure runs its logic next to the data, and the result comes back as rows, output values, or both. That arrangement fits well with the idea of a data API, where the procedure name and its parameters feel similar to a service call while table layouts and join details stay inside the database.

Many groups treat stored procedures as a boundary. Everything on the application side works with procedure names and parameter objects. Everything on the database side works with tables, indexes, constraints, and the procedure bodies that coordinate them. When that boundary stays stable, schema changes inside the database can evolve without constant edits to application SQL, because the procedure contract remains the same even as its internals change.

Calling Procedures From Application Code

Client libraries give Java code a way to call stored procedures through JDBC without building SQL strings by hand. JDBC offers CallableStatement for this purpose, which lets a developer prepare a call, bind input parameters, register output parameters, and then execute the stored routine. The driver turns that call into the actual CALL statement or vendor specific syntax that the database expects.

Let’s Imagine a SQL Server procedure named dbo.GetCustomerOrderSummary that returns recent orders for a customer and sets an output parameter for the total amount. Java code that calls it through JDBC can look like this:

The repository method treats the stored procedure as a named operation, with customerId going in and data for that customer coming back. The call string "{call dbo.GetCustomerOrderSummary(?, ?)}" declares two parameters that match the procedure signature. setInt binds the input value, and registerOutParameter asks for the total amount back from the database. Result rows move through the ResultSet loop, while the aggregate total arrives through the output parameter.

Many codebases wrap this pattern in a thinner adapter layer so that business code never touches JDBC types directly. That adapter maps procedure calls into higher level methods such as loadCustomerSummary, placeOrder, or archiveOldSessions. In that setup the data layer becomes a wrapper around a small set of procedures, and Java code sees those methods as its interface, while the stored routines carry all the SQL details.

PostgreSQL and MySQL work with Java in similar fashion. PostgreSQL often favors functions that return sets, so a call can go through a PreparedStatement with SQL such as SELECT * FROM get_recent_orders(?), while MySQL drivers frequently handle CALL GetRecentOrders(?) through CallableStatement. The application side keeps the same pattern of preparing a call, binding parameters, and reading rows or output values, even though the exact SQL string sent to the server varies between engines.

Performance For Bulk Workloads

Heavy workloads benefit when the database can perform large set operations in a single stored procedure call rather than a long stream of row by row commands from the application layer. Network latency, round trip counts, and repeated parsing all add up when a client sends many small commands. When a single procedure processes a large batch in one call, that overhead goes down and the optimizer can choose plans that handle the whole set in one go.

Billing runs make a good example. Suppose a data model holds accounts and usage records, and every month the system needs to generate invoices for every active account based on that usage. Java code could loop through every account and send an insert per invoice, but that burns network time and ties business throughput to a loop in application code. Stored procedures move that work closer to the data in a more set oriented way.

This T-SQL procedure posts monthly invoices for accounts within a given billing period:

This single procedure call scans the relevant usage rows, joins to the accounts table, groups by account, and inserts invoice rows in one set. Application code calls billing.PostMonthlyInvoices for a given billing period, passing the dates, instead of iterating through thousands of accounts on its own. The database engine can choose an execution plan that fits the entire set of usage rows, which tends to lower overhead compared to repeating smaller queries.

Java can treat this bulk operation as a single method call in its data layer. Take this for example that runs the billing procedure through JDBC:

The Java side now treats billing as one database operation that takes a period start and end date. All details about scanning usage, filtering active accounts, and inserting invoices live in the procedure body. Performance tuning around indexes, joins, and grouping logic stays in T-SQL, where database specialists can adjust it without changing the shape of the Java call.

Stored procedures also work well for other bulk cases, such as archiving old session rows into history tables, recalculating reward points across a customer base, or rebuilding summary tables for reports. In each of these cases, the client code sends a small parameter set while the database performs the heavy work in one or a few statements.

Security With Least Privilege

Relational databases give fine grained control over what each login can do, and stored procedures fit naturally with that model. Rather than granting an application user direct rights on tables, a security plan can grant only EXECUTE permission on a small group of stored procedures. Those procedures then act as gates that enforce business rules and log activity whenever data changes.

SQL Server makes this pattern fairly direct. An application login maps to a database user, and that user can receive EXECUTE rights on individual procedures while tables remain locked down. This setup is common to see:

With this configuration the app_user account can run dbo.GetCustomerOrderSummary and billing.PostMonthlyInvoices, but cannot select from dbo.Orders, billing.Usage, or billing.Invoices directly unless more rights are granted later. In SQL Server, ownership chaining can skip permission checks on underlying objects when the procedure and the referenced objects share the same owner. In that case, the caller needs EXECUTE permission on the procedure, not direct permissions on the tables.

For PostgreSQL, you can create a database account with limited rights, then grant execution on the functions and procedures that form the data API:

Granting EXECUTE on routines limits what can be invoked, but it does not by itself bypass table privileges in PostgreSQL. By default, functions run as SECURITY INVOKER, so the caller still needs the required privileges on referenced tables. If you want a low privilege role to run table access through a controlled routine, define the routine as SECURITY DEFINER under a trusted owner and lock down its search_path, or explicitly grant the needed table privileges.

This model also reduces the impact of SQL injection bugs in higher layers. Parameter binding with CallableStatement or prepared statements sends values separately from the SQL or procedure text. An attacker who finds a way to manipulate strings in application code still has to get through the stored procedure contract. If the only option is to call something like billing.PostMonthlyInvoices with different dates, there is far less room to smuggle in arbitrary data access.

Version Control For Stored Procedures

Database code benefits from the same discipline as application code, and stored procedures sit right at the center of that. Treating procedure bodies as first class source files brings them into code review, pull requests, testing, and release pipelines. Rather than editing procedure text directly in a management console, many shops keep a folder of .sql files in a Git repository and apply them through database tooling.

SQL Server Data Tools integrates database schema management with Visual Studio. Stored procedures, tables, and views live as files inside a database project. A developer edits dbo.GetCustomerOrderSummary.sql, commits the change, and shares it through normal source control. SSDT can compare the project state to a target database and generate upgrade scripts, which include ALTER PROCEDURE statements that bring the server in line with the project. That workflow treats stored procedures as part of the same lifecycle as the rest of the codebase, with history and review for every change.

Migration tools such as Flyway and Liquibase take a slightly different route. They focus on ordered migration scripts that describe how a database moves from one version to the next. Stored procedure changes become part of these migrations. In a Flyway setup, a new procedure or change to an existing one goes into a migration file with a versioned name such as V005__update_customer_summary_proc.sql:

Flyway records that this script has run on a given database by writing into its metadata table. When a new environment comes online, Flyway walks through the migration directory in order and applies any scripts that have not run yet. Stored procedure changes travel with the rest of the schema changes, which keeps development, test, and production databases in sync.

Liquibase supports a similar story through change logs that can be written in XML, YAML, JSON, or raw SQL. Change sets in Liquibase can include an sqlFile entry that points at a stored procedure definition, and Liquibase applies that file as part of a numbered change. Either way the important point is that the text of the procedure lives in source control, with a recorded history, rather than only inside the database catalog.

Treating stored procedures as versioned artifacts brings stability to the data API idea. The procedure signature and behavior are documented by the source file and by tests, changes pass through review like any other code, and rollbacks stay possible by reapplying an older migration or reverting in Git and redeploying. That mindset keeps the interface between Java code and the database predictable, even as schema and performance tuning evolve inside the stored procedures.

Conclusion

Stored procedures anchor business logic in the database, tying together parameter handling, compiled plans, and result structures in one named unit. They give application code stable entry points that behave like a data API while the database handles joins, filtering, and set based updates near the tables. Performance gains show up when heavy batches move into set oriented procedures, and security improves when application accounts receive only EXECUTE rights instead of direct table access. Version controlled procedure definitions and scripted migrations round out the idea by keeping that data API reproducible across environments and traceable as it changes.

1. SQL Server Stored Procedures Documentation

2. PostgreSQL CREATE FUNCTION Reference

3. PostgreSQL CREATE PROCEDURE Reference

4. MySQL CREATE PROCEDURE Reference

5. JDBC CallableStatement Javadoc

6. Flyway Database Migrations Documentation

7. Liquibase Documentation

Share Alexander Obregon's Substack

Developers use temporary tables in SQL to store intermediate results without touching permanent schema, which helps break large queries into stages and keeps report logic readable. Database engines provide their own flavors of temporary storage, yet they share common ideas such as session scope, automatic cleanup, and basic CREATE and DROP syntax. That view of how temporary tables behave in current SQL engines, how to create and remove them, and how to pass partial results through them for multi step reports helps explain how they keep large queries from turning into a maze.

Subscribe now

How Temporary Tables Behave In Databases

Temporary tables sit between short lived query results and permanent tables stored in the catalog. They use normal SQL syntax, yet the database keeps them separate from regular objects and ties them to a session or connection. That combination of familiar commands and special handling affects scope, lifetime, storage location, and performance, so it helps to walk through how the main engines treat them before building workflows on top.

Scope, Lifetime, Storage

Session scope is the first thing to keep in mind. MySQL, PostgreSQL, SQL Server, and Oracle all keep temporary data private in some way, but they do it with slightly different rules and naming schemes.

MySQL attaches a temporary table to the session that created it. No other connection can see it, and the name can be reused by other sessions with no conflict. The table vanishes automatically when the session ends, even if no DROP statement runs. The basic form looks like this:

That structure exists only for the current connection, and any attempt from another connection to query temp_daily_totals raises an error, even if the same schema and user name are involved.

PostgreSQL also ties temporary tables to sessions, but adds more detailed control around when rows disappear. The default behavior drops temporary tables at session end. With ON COMMIT clauses, the table can be kept while rows are removed, or the table can be removed entirely at transaction boundaries. One common form looks like this:

That definition keeps the table name and columns around for the whole session, but clears all rows whenever a transaction commits. Short lived working sets benefit from this because a new transaction starts with an empty table, while code does not need to recreate the structure each time.

SQL Server takes a different route and uses naming rules to flag temporary tables. Names that start with a single hash mark such as #recent_orders create a local temporary table visible only inside that connection and any nested scopes spawned from it. Names that start with two hash marks such as ##scratchpad create tables that can be shared by multiple connections while at least one connection still references them. Both live in the tempdb database behind the scenes. Common usage in code looks like this:

That table disappears when the connection that created it closes, so long running application processes that reuse connections tend to drop it explicitly once they no longer need it.

Oracle exposes global temporary tables, which work a bit differently. The table definition is permanent and stored in the data dictionary, so any session can reference the table by name. Data itself is still temporary, and each session sees only its own rows. The ON COMMIT clause controls when those rows go away. Typical usage looks like this:

Rows inserted into temp_order_buffer vanish on commit, while the table definition stays in place for reuse. Oracle stores the data in a temporary tablespace and isolates it from permanent tables in system catalogs.

Storage layout also matters. SQL Server places all temporary tables and internal work tables inside tempdb, along with temporary indexes and other supporting objects. Heavy use of temporary tables places load on that database, so administrators watch its size and I/O behavior. MySQL maintains internal temporary tables that may stay in memory or spill to disk depending on size and query features, with in memory internal temporary tables using TempTable by default (or MEMORY if configured) and on disk internal temporary tables using InnoDB. User created temporary tables use the engine given in the CREATE TEMPORARY TABLE statement, or the server default_tmp_storage_engine setting when no engine is specified. PostgreSQL keeps temporary tables in a per session schema named pg_temp and manages data in its shared buffer pool and temporary files. Oracle uses temporary tablespaces that can live on separate storage from permanent datafiles, which helps balance workloads.

Lifetime then depends both on scope rules and on the storage engine. A temporary table can vanish at commit time, session end, or connection close. Some engines drop structure and data in one step, while others keep the structure in the catalog and only purge data. Applications that create many temporary tables benefit from knowing when their databases reclaim these objects so they do not accidentally keep long lived connections full of leftover scratch tables.

Creation Syntax Across Popular Engines

Creation syntax stays close to normal CREATE TABLE commands, with small twists in each engine. The main difference is how to mark a table as temporary and how much control the command grants over commit behavior.

MySQL uses the TEMPORARY keyword. The table definition can be written from scratch, or a query can feed the definition through CREATE TEMPORARY TABLE AS SELECT. Both styles appear in production code, and the choice comes down to how much control is needed over column types and indexes. Direct table definitions often look like this:

And a later statement can load data into that structure through an INSERT from a query. When column types do not need adjustments, CREATE TEMPORARY TABLE can be driven directly from a SELECT:

PostgreSQL uses TEMP or TEMPORARY keywords and supports the same two styles. It also adds the ON COMMIT options that were mentioned earlier, so structure and data lifetime are tuned in the definition. Query driven forms often look like this:

That statement creates the table, copies in the current unpaid invoices, and arranges for the table itself to disappear when the transaction commits. For longer lived scratch tables, ON COMMIT PRESERVE ROWS keeps data until session end. Indexes and constraints are allowed as well:

SQL Server uses regular CREATE TABLE syntax, with the table name carrying all the temporary behavior through the hash prefix. Small differences still appear, such as the database automatically placing the object in tempdb and adding an internal suffix to guarantee uniqueness. Many scripts start with a definition and follow with an insert:

The #session_events table is session scoped. Statements run through sp_executesql in the same session can read it when it was created in the outer batch. If a # table is created inside a dynamic SQL string, later outer statements usually won’t see it, so scripts normally create the table first and then fill it from dynamic SQL when needed. A similar form with ##session_events builds a global temporary table that any connection can read while it exists.

Oracle treats global temporary tables more like regular tables that just happen to store transient data. Definitions are created once and kept in the schema. Sessions reuse them in the same way they reuse permanent tables, but Oracle quietly isolates rows per session. This definition is common to see in practice:

Later sessions run INSERT and SELECT against temp_session_payments as needed, and the database handles lifecycle of the rows based on the ON COMMIT clause and session end. Data dictionaries will list this table alongside permanent tables, although the underlying storage is managed in temporary tablespaces.

Throughout these engines, the CREATE syntax for temporary tables stays close to permanent table definitions, which keeps the learning curve gentle. The main differences lie in keywords, naming conventions, and commit options, while the mechanics of column definitions, constraints, indexes, and insert statements follow familiar rules.

Practical Workflows With Temporary Tables

Temporary tables turn long, tangled queries into a series of shorter steps that are easier to reason about. Instead of cramming every join, filter, and aggregation into one statement, the work can be broken into phases that pass data through temporary tables. That same idea applies to reports, stored procedures, and ad hoc analysis in query tools, with each engine bringing its own syntax but following the same broad workflow.

Breaking Large Reports Into Stages

Large reporting queries tend to do several things within a single statement. Monthly revenue reports, for instance, may filter by date, apply currency conversion, join to customer data, and aggregate by region and product category. When all of that lives in one SELECT, small changes become hard to manage. Temporary tables provide a scratch area where each stage can store its output before the next stage picks it up.

One way to handle this in PostgreSQL is to create a temporary table for the filtered and normalized base set, then another for enriched data, then run the final aggregation. The first step can filter sales and handle currency conversion:

This temporary table now holds only the rows and columns needed for the report, with values already converted to a single currency. Later steps no longer need to repeat conversion logic or scan dates outside that window.

The next step can enrich those rows with dimension data, such as customer region and product line:

At this point, the temp_enriched_sales table contains data that is ready to aggregate in different ways. One query can group by region and product line for a high level view:

Different queries in the same session can reuse temp_enriched_sales to focus on a single region or compare a few product lines without rejoining everything. Temporary tables give the report a middle layer that can support several final queries that share the same prepared data.

The same idea applies in MySQL with CREATE TEMPORARY TABLE and a query driven form. One filtering and conversion step can look like this:

Later statements in that session can join temp_january_sales to customer and product tables, or aggregate by different groupings, without scanning unrelated months again. Each engine handles the storage of the temporary data in its own layer, but the workflow of breaking the report into stages looks similar across platforms.

Passing Intermediate Results Between Steps

Workflows that involve several statements need a way to hand results from one statement to another without materializing permanent tables. Temporary tables are well suited for that handoff because they survive across statements in the same session and then vanish when the work is done.

Stored procedures in SQL Server use this frequently when several queries share the same filtered base set. Suppose a reporting procedure needs to filter a large audit log down to one customer for a date range, then use that same subset to produce multiple summaries. In that case, a temporary table fits right in:

The temporary table #base_events now carries only the rows relevant to that customer and date window. Any later query inside the procedure can reference it without repeating the filters on AuditLog.

Later, a step can summarize activity by day:

The next query in the same procedure can focus on event types:

Both summaries reuse the same filtered base set, which cuts down on repeated work and keeps the procedure logic easier to read. At the end of the procedure the temporary table can be dropped to keep the connection free of leftover objects:

MySQL routines can adopt a similar practice with CREATE TEMPORARY TABLE. The session can call a procedure that creates a temporary table at the start, populates it, then runs several queries against it. That table stays private to the connection that owns the procedure call, and MySQL removes it when the session finishes, so the procedure body does not need permanent staging tables for intermediate steps.

PostgreSQL functions can also lean on temporary tables, although many workflows there prefer common table expressions or materialized views. When a function needs to write and read the same intermediate data across multiple statements, a temporary table in the same session still provides that bridge. After creation, the table sits in the pg_temp schema for that session, and later queries in the function can refer to it directly until the function drops it or the session ends.

Dropping Temporary Tables At The End

Automatic cleanup rules for temporary tables handle many cases, but explicit drops still help keep long lived sessions tidy and avoid surprises in shared environments. Connection pools in application servers are a good example. One pooled connection can serve many different requests over time, and each request may create its own temporary tables. Dropping those tables at the end of a workflow frees internal resources and reduces the risk of name clashes when another request runs later on the same connection.

Development and query tools benefit from explicit cleanup as well. When a script is run repeatedly in a tool like psql, sqlcmd, or a graphical client, leftover temporary tables from a previous run can cause errors if the script tries to create a table with the same name again. Adding DROP statements near the end of the script, or guarded drops before the CREATE statements, keeps the session in a known state and removes the need for manual cleanup between runs.

Different engines offer their own syntax for drop operations. MySQL supports DROP TEMPORARY TABLE and an IF EXISTS option so that scripts can remove tables without raising errors if those tables have already gone away. PostgreSQL uses standard DROP TABLE, also with IF EXISTS, and applies it to temporary tables in the same session in the same way as permanent ones. SQL Server supports DROP TABLE for # and ## tables and lets scripts guard the operation with DROP TABLE IF EXISTS. Oracle global temporary tables can be cleared with TRUNCATE TABLE when the structure should remain for reuse, or removed with DROP TABLE when the definition should go away as well.

Global temporary tables in SQL Server deserve special attention because they are visible across connections while any connection still references them. If DROP TABLE ##scratchpad is forgotten, shared staging data can remain in place and confuse later sessions that read from that name. Dropping global temporary tables as soon as shared work finishes prevents that bleed over and keeps the shared namespace predictable.

Temporary tables give SQL workflows a flexible middle ground between one shot result sets and permanent schema, but they share resources with other parts of the database engine. Creating them thoughtfully and dropping them when their job is done keeps those workflows efficient and avoids side effects in pooled or long running sessions.

Conclusion

Temporary tables give SQL developers controlled, short lived storage between single query results and permanent schema, with scope and lifetime governed by engine specific rules. MySQL and PostgreSQL attach temporary tables to sessions, SQL Server binds local and global temporary tables to connections through hash prefixed names in tempdb, and Oracle treats global temporary tables as permanent definitions with transient per session rows. In all of these engines, temporary tables rely on standard CREATE and DROP statements while acting as a staging layer where intermediate rows are written, queried by later statements in the same session, and then removed through automatic cleanup or explicit drop commands.

1. MySQL CREATE TEMPORARY TABLE Documentation

2. PostgreSQL CREATE TABLE Temporary Relations

3. SQL Server CREATE TABLE And Temporary Tables

4. Oracle Global Temporary Tables Reference

   Share Alexander Obregon's Substack

RLS, or Row level security, lets a database keep many users or tenants in the same tables while still restricting which rows each session can read or change. Instead of putting every access rule in application code, the database applies extra predicates whenever a query touches a protected table. That arrangement creates a single place for row visibility rules, which matters for multi tenant products, compliance requirements, and shared reporting environments.

Subscribe now

Row Level Security Basics

Modern database servers started with permissions that act at the level of whole tables, views, and schemas. Row level security adds an extra check so the server can decide not only which table a session may query but also which individual rows stay visible for that session. RLS can keep several tenants or groups inside shared tables while still keeping their data separated from one another, even when every query flows through the same application logic. Traditional privilege systems answer questions like “can this login run SELECT on table orders”. Row level security adds a second question about row visibility that runs after normal privileges pass. Any session that has table level SELECT still only sees rows that satisfy the row security predicate. That extra filter comes from policy rules that reference both columns on the table and information about the current user or tenant.

What Row Level Security Does

Many developers first meet row level security while working on a multi tenant product where every tenant has its own users, orders, and reports, but all of that data lives in shared tables. Granting plain SELECT on those tables would expose every row to anyone with that permission, so the database needs a rule that trims the result set for each login without asking every query author to remember the right WHERE clause.

Standard SQL permissions treat table access as all or nothing. Either the session has SELECT on a table or it does not, and the server does not inspect which specific rows belong to that caller. With row level security turned on for a table, the server evaluates a policy whenever a statement touches that table. The policy behaves like an extra WHERE expression that must be true for a row to participate in SELECT, UPDATE, or DELETE for that user.

One way to think about this that can help beginners including me when I was learning this, is to think of the database rewriting plain queries so they always include a filter that reflects the current user. Take this example with a shared order:

Ordinary privileges say whether that statement is allowed at all. Row level security then trims rows down to only the ones that match a user or tenant rule. The effect resembles running a second query with a filter that references some identity column.

No real system literally rewrites the text of the query in that way, but the final result set tells the same story. Rows that do not satisfy the policy condition vanish for that session, and aggregates such as counts and sums naturally reflect only the visible subset.

Row level security also applies to write operations in engines that support it. Updates and deletes normally see only rows that pass their policies, so attempts to update someone else’s row in a shared table simply affect zero rows instead of crossing tenant boundaries. Some platforms also enforce checks on inserted rows so new data must satisfy a policy before it enters the table.

Any table can have row level security active or inactive as a feature flag. Databases with built in RLS usually expose a DDL switch that turns it on. A family of systems uses a statement in the same idea:

After that change, every ordinary query that references the employee table flows through the row security machinery, and the table behaves as protected data rather than freely readable data.

Row Level Security Rules

Rules that implement row level security include several moving parts that work in combination during query execution. Treating those parts separately helps explain how databases answer questions about which rows belong to which caller.

Every rule starts with a table that needs protection. Some platforms mark this on the table itself, such as a property that says row level security is active. Others attach the rule through a security object that names both the table and a predicate. In all of those cases, the database records that row level decisions must run for that table in addition to the normal privilege checks.

Predicates carry most of the logic. The predicate is an expression that returns true or false for a row. It may read columns on the row, like tenant_id or owner_id, and values supplied by the current session. Engines that support RLS usually store this predicate as part of a policy definition, which keeps the rule close to the table it guards.

Many systems phrase a predicate directly in SQL. Policies in those systems frequently look like short WHERE clauses that reference both a column and some function call that reflects who is connected.

That single expression can sit inside a policy, a view, or a stored function, yet the core idea stays the same, only rows with a matching tenant id remain visible.

Other vendors let administrators write policy functions in a procedural language like PL/pgSQL, T-SQL, or PL/SQL that return a boolean value or a text predicate. Those functions still end up controlling a WHERE condition, but the extra language features make it easier to handle complex cases, such as administrators who should see multiple tenants or audit robots that should see every row.

Rules also need a scope so the server knows which statement types they apply to. Some policies only apply to reads and leave writes to application code, while others apply to both reads and writes so those queries stay inside the same boundaries. Typical configurations can permit users to read only their own rows yet allow administrators with a specific flag to bypass RLS or rely on a broader predicate.

Tenant isolation offers a good starting point when talking about rule pieces. In a shared billing table, every row carries a tenant identifier, and the predicate compares that column to a tenant id stored in session state. Take this example in SQL that focuses on the table structure first:

Row level security never requires a special primary identifier, but many real tables that participate in RLS still have one so updates and joins stay fast and precise.

With a table like that in place, a policy only has to express a comparison between tenant_id and some tenant marker attached to the current session. Different engines spell that policy in their own syntax, yet the logical core is always a predicate along the same lines as the earlier tenant_id expression.

User Context In Session Data

Row level security always needs some notion of who is asking. That identity can come from the database login, a mapped application user id, a tenant id, or a blend of all three. Whatever form it takes, that data has to be available inside the predicate so the database can compare it to the values stored in rows.

Database servers expose several ways to represent identity. Most have a base concept of current user, which refers to the database account behind the session. Many also offer configuration parameters or session context slots that an application can populate just after opening a connection. Those values stay tied to the session and can be read back from inside SQL or policy functions.

One common arrangement for multi tenant web applications ties HTTP authentication to database context. Middleware code decodes a session token, looks up a tenant id and application user id, and then records those values through small statements at the start of every transaction.

With that information stored as session state, predicates on protected tables can call functions that read those settings. An expression such as tenant_id = current_setting('app.current_tenant') will only pass rows that belong to the current tenant, no matter which query text the application sends later in the transaction.

Vendors ship different tools for this idea, but the template stays similar. Some platforms use special context functions that read from a namespace of session variables, while others rely on explicit session context features that store values under string names and expose them to predicate logic. Regardless of the exact feature name, row level security depends on that bridge between user or tenant identity in the session and matching columns in protected tables.

Systems that lack built in row level security features still rely on user context in a similar way. Views and stored procedures can read CURRENT_USER or other identity markers and use those values inside WHERE clauses that restrict rows. The main difference is that the enforcement lives in views and procedures rather than in a dedicated policy system tied directly to each table.

Row Level Security In Practice

Projects rarely work with row level security as an abstract feature. Real value shows up when a real world database engine takes the general idea of policies and user context and turns that into specific DDL statements, functions, and execution rules. Different vendors picked their own mechanics, but they all revolve around three ingredients. There is a way to mark which tables should be protected, there are predicate rules tied to those tables, and there is some channel that feeds user or tenant identity into those rules at query time.

PostgreSQL, SQL Server, Oracle Database, and MySQL all sit in this picture, although they do not offer the same feature set. PostgreSQL and SQL Server ship built in row level security features. Oracle Database provides Virtual Private Database policies that behave in a closely related way.

PostgreSQL Policies

PostgreSQL keeps row level security turned off for a table until the owner makes an explicit change. That switch prevents surprises on existing schemas. When RLS is off, queries only pay attention to normal privileges such as SELECT or UPDATE on the table. After RLS is turned on, the server evaluates policies for every row that a statement touches, and only rows that pass those policies participate in the result or write operation.

The feature starts with a table. Let’s say we are working with a billing application in Eau Claire that tracks customer accounts in a single shared table with one tenant identifier per row:

PostgreSQL does not treat that table as row protected yet. Turning on RLS takes one DDL statement.

After that change, queries from ordinary roles only see rows that pass at least one policy. When no policy matches, PostgreSQL treats the table as if it had zero visible rows for that user. Superusers and roles with the BYPASSRLS attribute are exempt, and the table owner can bypass RLS as well by default, which helps administrators inspect data when they need to debug policies.

Policies carry the row filter logic. PostgreSQL stores them as named objects tied to a specific table. Policies can apply to all commands or to just one of SELECT, INSERT, UPDATE, or DELETE, and they can be restricted to certain roles.

That policy instructs PostgreSQL to keep rows where tenant_id matches whatever value the application put into the app.current_tenant setting for that session. Any other row in the accounts table behaves as if it does not exist when a SELECT runs through that connection.

Policies can also check new rows that an INSERT or UPDATE tries to write. The WITH CHECK expression covers this case. If it is absent, PostgreSQL reuses the USING expression, which keeps read and write rules consistent.

With those two policies active, tenants read only their own rows and can only insert or update rows that match the tenant id stored in their session setting. Any attempt to sneak in a row with a different tenant id fails at the database level, even if application code forgets to enforce the rule.

PostgreSQL also distinguishes between permissive and restrictive policies. Permissive policies broaden access, and the server combines their predicates with logical OR. Restrictive policies narrow access, and their predicates combine with logical AND. Each row must satisfy at least one permissive policy and all restrictive policies to appear in query results. This split makes it possible to have a general tenant wide policy, then an extra restrictive policy that hides rows with deleted_at set or that limit certain roles to older data while administrators see more.

SQL Server Security Policy

Microsoft SQL Server introduced row level security through a feature that centers on predicate functions and security policies. Instead of embedding the predicate directly into the table definition, SQL Server expects an inline table valued function that checks one row at a time, then a security policy that attaches that function to a table.

Predicate functions live in a schema, just like tables and views. They must use the SCHEMABINDING option so that the engine can reason about dependencies. This next predicate ties customers to their own rows by tenant id:

That function reads from SESSION_CONTEXT, which acts as a key value store bound to the current session. Application code sets the value with sp_set_session_context when a connection opens or when a request starts, and the predicate function compares that stored value to the row’s tenant identifier.

Tables that need row filtering can then reference this function in a security policy:

After that policy is in place, SQL Server calls the predicate for rows touched by SELECT, UPDATE, and DELETE statements. Rows that fail the predicate do not appear in result sets and do not take part in those write operations. The calling code does not need to mention tenant restrictions at all, the policy applies to the table regardless of the query text.

Block predicates extend the model to prevent disallowed writes. A block predicate that targets INSERT, UPDATE, or DELETE can stop operations that would have introduced out of tenant rows or modified them in ways that break security rules. Both filter and block predicates share the same table and rely on predicate functions, but they attach to different statement types through the security policy.

SQL Server enforces row filtering through security policies, and it restricts predicates per table by operation. If a table already has a predicate defined for a given operation, trying to add another predicate for that same operation results in an error. Those limits reduce the risk of conflicting logic where one predicate tries to hide rows that another predicate expects to see. Developers working on row level security in SQL Server usually end up with one or two small predicate functions that all related policies share across multiple tables.

Oracle Virtual Private Database

Oracle Database takes a slightly different route with its Virtual Private Database feature, or VPD. Instead of inline functions that return a boolean column, VPD policy functions return a text string that represents a predicate. The engine then appends that predicate to user queries as if they contained that condition from the start.

Administration begins with a normal table. Let’s say we are working with a human resources schema named HR that keeps employee data in a single table that serves multiple departments:

Virtual Private Database relies on the DBMS_RLS package to attach policies. Before that call, a policy function has to exist. That function reads context values, then assembles a predicate string such as dept_code = 'SALES':

The call to DBMS_RLS.ADD_POLICY links this function to the table:

With that policy in place, Oracle Database consults the HR_APP_CTX context for a DEPT_CODE value during query parsing. The returned predicate string attaches to user queries that reference HR.HR_EMPLOYEES for the listed statement types. Queries that do not mention dept_code in their text still receive that filter at runtime, so users only see rows from their own department. VPD also supports column level policies. Policy rules can apply only when certain columns appear in the query, which helps when columns such as salary need tighter access while columns such as full_name are fine to expose more broadly. In all cases, the engine enforces access by attaching a dynamic predicate to the statement issued against the protected object.

Context values for VPD policies usually come from SYS_CONTEXT. Namespaces such as USERENV expose generic information like session user or client identifier, and custom namespaces created by the application hold values such as department or tenant identifiers. Policy functions stay relatively small because they mainly translate those context values into predicate strings that Oracle attaches to SQL statements.

MySQL Views With Triggers

Standard MySQL editions do not ship a dedicated row level security feature at the server level. Access control still revolves around object privileges, so a user with SELECT on a table can see all rows in that table. Many installations still need finer grained control, so administrators build row restricted access paths with views and grants, sometimes helped by triggers and stored routines.

One common arrangement relies on a view that filters rows based on the current MySQL user. The base table may carry an owner column that matches the MySQL account string returned by USER():

The filtered view then narrows access:

Grants direct users toward the view and away from the base table:

With that structure, the user alex interacts only with current_user_notes. Any query that user sends against the view inherits the WHERE owner_name = USER() filter, and the absence of privileges on project_notes prevents bypassing the filter through direct table access.

Triggers and stored procedures can reinforce that structure for write operations. An INSERT trigger on project_notes can force owner_name to match USER() or SESSION_USER() regardless of what value the client supplied, and an UPDATE trigger can reject attempts to change owner_name to someone else. Some MySQL based systems also centralize writes through stored procedures that check USER() against ownership rules before allowing inserts or updates to proceed.

Cloud platforms built on MySQL, such as Amazon Aurora MySQL and managed RDS for MySQL, document similar setups. Application code connects with accounts that only see filtered views and that rely on triggers or stored routines for data changes. Even though there is no single statement that turns on row level security for a table, the combined effect of views, grants, triggers, and session user checks gives row based isolation that behaves much like built in RLS in day to day work.

Conclusion

Row level security pushes row filtering into the database engine by tying policy predicates to tables and driving those predicates with user or tenant context on every query. PostgreSQL does this with table policies and current_setting, SQL Server with predicate functions and security policies, Oracle Database with VPD functions and DBMS_RLS, and MySQL through views, grants, and triggers around CURRENT_USER. These mechanics define which rows a session can read or modify and keep shared tables isolated while application code continues to send ordinary queries without tenant specific filters in every statement.

1. PostgreSQL Row Level Security Documentation

2. SQL Server Row Level Security Guide

3. Oracle Virtual Private Database Guide

4. MySQL Views and Privileges Reference

5. Aurora MySQL Row Level Security Example

Share Alexander Obregon's Substack

Production databases tend to carry queries that feel slow even on strong hardware, and index hints give database engineers a direct way to override the optimizer when its choices do not match the actual data distribution or traffic behavior, so engineers can steer specific statements toward index plans that trim I O and response time.

Subscribe now

How Index Hints Affect Query Plans

Index hints sit inside query text and tell the optimizer how to treat specific indexes. Without hints, the engine reads statistics, builds several candidate plans, and picks the one that looks cheapest in terms of I O and CPU cost. With hints, that search space narrows, because certain indexes are pushed to the front or removed from the options entirely. MySQL and SQL Server both follow this broad idea, even though their hint syntax and details differ.

These hints do not turn off the optimizer. The engine still has to decide join order, join algorithms, sort strategies, and many other details. Hints mainly control which access routes are allowed for a table, such as a full scan, a seek on an index, or a scan of a covering index. That makes them a precise tool for queries where the planner repeatedly picks an index that works poorly with real data statistics.

What Query Optimizers Do

Database engines treat a text query as a starting point for building an execution plan. The first step turns the string into a parsed tree, which records tables, columns, predicates, and join conditions in a structured form. That tree may expand views and inlineable subqueries so that the optimizer can reason about a single, unified plan instead of many smaller fragments. After parsing, the optimizer reads metadata. It checks which indexes exist on each table, what columns those indexes contain, and what kind of statistics are available on base tables and indexes. Histograms and density values stored with statistics give the planner a way to estimate how many rows match a predicate such as status = 'ACTIVE' or order_date >= '2026-01-01'. Those row count estimates drive almost every later decision, because they determine how much work each part of the plan must do.

When metadata and statistics are in hand, the planner starts building alternative plans. One candidate may scan an entire table and apply a filter to every row, while other candidates seek into selective indexes and touch only a small slice of the data. Further variation appears around join order and join type. The engine can join customers to orders in more than one sequence, and it can choose nested loops, hash joins, or merge joins based on estimated row counts and sort properties.

Indexes enter this process as access options for a table. One table that has three secondary indexes and a clustered index presents several possible access patterns, including a full scan of the clustered index and different seek or scan patterns on each secondary index. Without hints, the optimizer weighs all of them and keeps the ones that look most promising. With hints, that list shrinks, which guides the final plan toward specific indexes.

MySQL offers hints such as USE INDEX, FORCE INDEX, and IGNORE INDEX inside the FROM clause. These apply to SELECT and UPDATE statements and to multi table DELETE statements, and they can also be scoped to phases like join planning, ordering, or grouping. Newer MySQL releases also document index level optimizer hints meant to replace these, with USE INDEX, FORCE INDEX, and IGNORE INDEX expected to be deprecated in a future release. Here is a small example with USE INDEX:

This query still goes through normal planning, but the hint tells MySQL to favor idx_shop_orders_customer for access to shop_orders and to push other indexes to the side for this statement.

SQL Server applies table hints by placing WITH (hint options) immediately after the table name. One common form is the INDEX hint that directs planning to a specific index identifier or name:

That hint does not fully lock down the plan, yet it steers the optimizer toward IX_Orders_CustomerId when it decides how to read Sales.Orders. Other parts of the plan, such as join choices and sort operations, still come from the usual cost based process.

Index Selection With Statistics

Statistics sit at the core of index choice. Each index and base table can have statistics objects that record how values are spread across the column domain. Histograms on columns such as status can reveal that only a small fraction of rows use a value like 'PENDING_REVIEW'. Density values for a composite index on (customer_id, order_date) can show how many different combinations exist in the table. Planners rely heavily on these numbers when they decide which index to use.

When a predicate references columns with up to date statistics, the engine can estimate selectivity for each index that covers those columns. If a filter on customer_id and order_date is expected to match only a small band of rows, a composite index on those columns looks attractive. If statistics indicate that almost every row matches a predicate, the planner may prefer a scan, because seeking into an index would just add overhead without much benefit.

Problems appear when statistics no longer match reality. Data can change dramatically after a bulk load, seasonal spike, or archival process. Skewed distributions can also confuse the planner. Think about a flag column with values that are almost always zero, apart from a tiny active slice. Queries that focus on the rare slice can perform well with a selective index, while queries that touch the common slice behave badly if the engine misjudges how many rows will match.

Let’s take a look at a MySQL example of this, say we are working with a logging table that holds web events:

Suppose most rows have event_type = 'PAGE_VIEW', while rare events such as event_type = 'PURCHASE' form a tiny minority. One query that hunts for recent purchase events by a single user can be:

If statistics on event_type and user_id do not reflect the true distribution, MySQL may misjudge which index will touch fewer rows. The planner may pick idx_page_events_type_time when that index leads to many more reads than a targeted seek on idx_page_events_user_time for this query shape. Index hint use can nudge the engine toward the composite index on user_id and event_time.

That hint tells MySQL to center its plan on idx_page_events_user_time for access to page_events. The predicate on event_type still applies, but now it filters rows that already passed the user_id and event_time filter driven by the composite index.

SQL Server runs into the same category of problem. Say there is a table for support tickets:

Think about a report that focuses on open tickets for a single account in a recent date range:

Depending on statistics, SQL Server may gravitate toward IX_Tickets_Status_Priority and then apply the AccountId filter as a residual predicate. If that index leads to a large scan, a table hint can direct the engine back toward the composite index centered on AccountId:

This version tells the optimizer to base access on IX_Tickets_Account_CreatedAt. The statistics on that index still matter, but the hint removes other indexes from the competition for this table inside this query.

Risks Of Forcing Index Choices

Hints that steer index choice help with stubborn slow queries, yet they bring tradeoffs that are easy to overlook. Every hint hard codes an assumption about data distribution, workload habits, and index layout. As months pass, tables grow, data skews change, and administrators add or drop indexes. That hint can later turn into a liability when those assumptions no longer hold. Limited flexibility is one of the biggest side effects. Without hints, the optimizer can generate different plans for different parameter values, storage layouts, or statistics states. It can pick a selective index for rare values and a scan for common ones. With a forced index, that freedom shrinks. The planner must keep the hinted index in the plan, even for parameter values that would benefit from a different route through the table.

Schema changes get more awkward when index hints spread through application code, because dropping or renaming a hinted index affects not only the optimizer’s choices but can also cause queries to fail at parse time, forcing database engineers to comb through code bases and stored procedures to update or remove those hints before they can safely adjust index definitions.

Say we work with a customer table in MySQL that once relied on a narrow index on last_name to support specific reports:

At some point new reporting needs lead to a wider composite index.

If an early version of the reporting query had a hint like this:

And later the engineering group decides to drop idx_customers_last_name because idx_customers_last_name_city serves modern reporting better, that hint turns into a problem. Dropping the index without touching the query causes MySQL to raise an error, because the hint now references an index that no longer exists.

SQL Server faces similar issues with long lived table hints. Index hint use that names a specific index id or name needs a code change whenever index definitions move. Hint heavy code bases tend to slow down index tuning work, because every change requires coordination between database administrators and application developers.

For these reasons many engineering groups reserve index hints for specific pain points, document them carefully, and revisit them during major schema or workload changes. Better index choices, fresher statistics, and query rewrites usually carry less long term risk than a broad wave of forced index selection across an entire application.

Hands On Index Hint Examples

Query tuning makes more sense when you see the shape of an actual table, an actual query, and the plan that the optimizer picks. Index hints come into play when those plans stay slow even after you create reasonable indexes and let statistics update. The examples here stay focused on read queries, because those are common in reporting and web backends, and they give you a direct way to see how hints tie into EXPLAIN output and execution plans.

MySQL Index Hint On A Read Query

You will often see MySQL applications track customer purchases in a table that grows every day. Indexes help keep recent history queries fast, but the optimizer still has choices that do not always line up with how the data is distributed. As one case, take this table that records orders from an online store:

Assume most traffic comes from screens that show recent orders for one customer at a time. A read query can look like this:

On a large table with a good composite index, a plan that seeks into idx_orders_customer_date, walks back through recent dates for that customer, and stops after 50 rows keeps I O relatively low. That path lines up with the filter and the sort order. On some data sets, though, MySQL estimates that many rows will match and quietly chooses a full table scan.

Running EXPLAIN on the same text makes that choice easier to see:

One run against a table with about a million rows can produce a row that reports type = ALL, rows = 1000000, and an Extra field that mentions a filesort. That combination means the engine walks the whole table, applies the filter to every row, sorts the survivors on order_date, and finally discards all but the first 50. On modest hardware that kind of plan can sit in the few hundred millisecond range whenever the buffer pool does not already hold the needed pages.

Index hints give you a way to push the optimizer toward the composite index. The hint becomes part of the query text and stays under version control with the rest of the application.

With that hint in place, EXPLAIN tends to report type = range or ref, the key column moves to idx_orders_customer_date, and the rows estimate drops sharply, sometimes into the low hundreds on realistic data. That change reflects a new access route where MySQL seeks directly into the slice of the index that matches customer_id = 1001 and the date range. The engine still has to read base table pages for the selected rows, yet it avoids touching the entire table and also avoids sorting all matching rows in memory or on disk.

On a basic test instance with data in that range, queries that follow the hinted plan can land in the tens of milliseconds, because they only read the portion of the index that satisfies the filter and sort requirements. Timings always depend on hardware and load, but the mechanical reason stays the same. The hint makes the optimizer treat the composite index as the primary route into orders and narrows the number of candidate plans that involve full scans.

MySQL also supports the reverse move, where you tell the optimizer to ignore one or more indexes that tend to drag plans in the wrong direction. Queries that suffer from an unhelpful status index can be written like this:

That change forces the planner to treat idx_orders_status as unavailable for the duration of that statement. The remaining choices still go through cost based planning, but the hint removes one distracting option from the search space, which can guide MySQL toward access paths built on idx_orders_customer_date without fully locking it into a single index name.

SQL Server Index Hint Through Table Hints

SQL Server tends to appear in environments that lean on reporting queries and mixed workloads, so table hints interact with a wider range of query plans. Index hints come through the WITH (hint options) clause after a table reference and can nudge the optimizer toward indexes that track business filters more closely than its default choice.

Take this Sales table in a transactional database that feeds dashboards and nightly reports:

Reporting screens that show recent completed orders for one customer pull from this table with a query similar to the MySQL example but written in T SQL:

An estimated execution plan in SQL Server Management Studio can reveal that the optimizer chooses an index seek on IX_Sales_Status_OrderDate, applies a residual predicate on CustomerId, and performs key lookups into the clustered index to fetch TotalAmount. When nearly every row has Status = 'Completed', this plan ends up touching a large portion of the table, because the first predicate in the composite index is not selective for this workload.

A table hint can steer the engine in a direction that fits the actual filter better. Placing an index hint right after the table name changes the planning step:

Execution plans generated from this text typically show an index seek on IX_Sales_Customer_OrderDate with a range on CustomerId and OrderDate, and then a predicate on Status applied to the index rows that pass the first filter. If the nonclustered index includes Status and TotalAmount as included columns, the engine can satisfy the entire query from the nonclustered index alone, without key lookups back to the clustered index.

On a medium sized table that holds millions of rows, that change can make a visible dent in query time, because the seek on a customer and date range greatly reduces the number of rows that participate in the plan. I O drops, and memory pressure from sorting or buffering rows falls as well, which benefits other queries that share the same instance.

SQL Server also exposes a FORCESEEK hint, which gives a finer tool for encouraging index seeks instead of scans. When a table has a wide index that the optimizer occasionally scans in full, even though a seek would work, FORCESEEK can channel the engine toward the seek operator:

That form keeps the access path centered on IX_Sales_Customer_OrderDate and makes it far less likely that SQL Server will choose a scan on that index for this query. Combined with included columns that cover the select list, the plan can stay small and predictable, which helps when the same query text runs across many tenants or accounts with different data volumes but similar data shapes.

Conclusion

Index hints give you a direct way to influence how MySQL and SQL Server read tables by specifying which indexes the optimizer treats as entry points and how those choices interact with statistics and predicates. With the mechanics and examples in this article, you can map a slow query to its plan, see when the planner leans on a scan or a weak index, and then apply targeted hints or indexing and statistics changes that keep execution plans aligned with the filters and sort orders that matter for real workloads.

1. MySQL 8.0 Reference Manual Index Hints

2. MySQL 8.0 Optimizer Statistics For InnoDB

3. SQL Server Query Hints Documentation

4. SQL Server Statistics Documentation

   Share Alexander Obregon's Substack

Text search in relational databases usually starts from a basic need to filter rows by a name, title, or description when a user types only part of a word or phrase. At a small scale, a filter with the LIKE operator can feel fast enough as queries read only a modest number of rows. As tables grow to millions of records, the same filter can force long scans across indexes or even the whole table if the search expression does not line up with how data is stored. Partial matches stay fast when the database pairs the right search operators, such as LIKE, full text search features, and trigram based extensions, with index structures that match the way text values are accessed.

Subscribe now

How Partial Text Search Works With Indexes

Partial text search sits where human friendly filters meet storage rules inside a database engine. Users sometimes type part of a product name, a piece of an email address, or a single word from a long description, and expect the application to bring back the right rows with only that fragment to work with.

Index structures decide how much data the engine has to read to answer that request. B tree indexes on text columns hold values in sorted order and let the engine jump to a position that matches a given value or a range of values. When conditions with LIKE match the way values are laid out in that index, searches stay close to index range scans. When a condition hides the prefix or wraps the column in a function, the planner loses that benefit and drifts toward scans that touch a large part of the table.

Basic Pattern Matching With LIKE

Many SQL queries that search text start off with the LIKE operator. This operator treats text as a sequence of characters and compares it to a template that can include wildcard symbols along with normal letters or digits. LIKE understands two wildcard characters. The percent sign % stands for any sequence of characters, including an empty string. The underscore _ stands for a single character. All other characters in the match string have to line up exactly, subject to collation and case sensitivity rules in the database.

Let’s look at a filter that looks up products by prefix and fits this model:

This condition has a constant prefix Post followed by a wildcard. On a B tree index built on products(name), many engines can treat this as a range that starts at Post and ends just before the next possible string after that prefix. That lets the engine scan only a slice of the index, not the entire table.

Suffix matches keep the wildcard at the front and fix the tail of the string instead:

This kind of match has no fixed beginning, so the engine cannot map it to a contiguous range in a B tree index on name. A common plan in that case is a full scan of the table or a scan of every entry in the index, with the LIKE comparison applied row by row.

Substring matches fall into the same category:

With wildcards on both sides, the database must examine many values to see which ones contain that middle fragment. A normal B tree index on name does not help much with that predicate.

The underscore wildcard matches a single character position and keeps the rest fixed:

This condition says that the first two characters must be A and B, the fourth through sixth characters must be 1, 2, and 3, and the third character can be anything. Index support here depends on the database engine and collation rules. Some engines can still use the fixed prefix AB to find a starting point in the index and then apply the full match expression as a filter on rows that share that prefix.

Real applications frequently combine LIKE with other predicates. Many customer search pages only allow queries inside one city or one status flag:

The equality on city narrows the set of rows that need to be checked, and the prefix Sm% can align with an index on (city, last_name) in that order. Index choices on multiple columns become important in those cases, because the leftmost column or columns in the index define how the B tree is laid out and what range of LIKE prefixes keep the scan narrow.

Limits Of Plain LIKE On Large Tables

Large tables change how LIKE behaves in practice. Filters that feel acceptable on a few thousand rows can turn into heavy operations on tens of millions of rows when the index cannot help.

Search strings that begin with % cause trouble first. Take this for an example:

Here, the code runs against a large catalog, the engine has no way to guess where names that end in Pro sit inside a sorted index. That usually leads to a plan that touches the full table or the full index and applies the LIKE check to every candidate value. With enough rows, that plan pushes latency up and increases load on the storage layer.

Computed expressions inside the predicate cause similar trouble. Many codebases want case insensitive search and reach for LOWER or UPPER calls:

Plain indexes on name no longer match that condition, because the stored index entries hold the original text while the predicate runs against lowercased text. Several engines, including PostgreSQL and Oracle, support indexes built on expressions such as LOWER(name) so that queries can use the transformed text directly in the index. When that index is not present, the engine falls back to a scan.

Function calls that trim whitespace or remove accents have the same effect. Any expression that changes the value the index sees makes a normal index on the base column less helpful unless a matching index on the expression exists.

Multiple wildcards in the middle of a search string increase the cost as well.

This asks the engine to find rows containing pro, then later search, then later guide, all in that order, without any help from an index on the original column. For long strings and large tables, that type of search scans many bytes per row.

Parameter driven search interfaces bring an extra challenge. One field can accept a prefix search, while another allows a contains search, depending on user input:

Query forms like this lead to different access patterns at runtime. Prefix search can work with an index, while contains search does not. Query planners inspect the constants or bound parameters at execution time and decide how much of the index can help, but parts of the filter that begin with a wildcard tend to fall back to scanning.

These limits pushed database vendors to add features geared toward indexed text search such as full text search and trigram style matching. Those features build on different index structures and search strategies and address cases where LIKE alone cannot keep partial text search fast on very large data sets.

Indexing Strategies For Partial Matches

When the index structure lines up with how the query engine reads text, partial text search becomes much more practical. Queries that only need prefixes can lean on ordinary B tree indexes, while word based search benefits from inverted indexes that map terms to rows. Substring search and fuzzy matches depend on structures that store short character fragments such as trigrams. All of these live side by side and address different types of text filters that appear in real applications.

Databases bring these strategies to life through specific index types and operators. B tree indexes back up LIKE 'prefix%' filters. Full text indexes support boolean and ranked search against large bodies of text. Trigram indexes and related features in extensions push partial and typo tolerant search toward index backed plans instead of table wide scans. Getting comfortable with how each one works makes it easier to pick the right option for a given search screen or API endpoint.

Prefix Search With B Tree Indexes

Many user interfaces ask for the beginning of a value rather than the full string. Autocomplete on customer last names, suggestions for city names, or product codes typed into a search box all fit this kind of prefix search. B tree indexes handle these cases well when queries keep the leading characters fixed and place wildcard symbols only at the end.

B trees store values in sorted order and arrange index entries in pages that connect through a tree of pointers. When a query looks for a range of values, the engine can follow the tree down to the first matching value and walk through pages in order until the range ends. Prefix searches match that behavior closely, because strings that share a prefix sit next to one another in the index.

This PostgreSQL example makes this a bit easier to see:

With that index in place, a query that matches a prefix has a good chance of using it:

The planner can translate LIKE 'Sm%' into a range that starts just before Sm and ends just before the next possible prefix after Sm. That plan scans only the range of index entries whose last_name values begin with Sm and then retrieves the matching rows from the table.

MySQL and SQL Server apply the same basic idea to LIKE 'prefix%' on indexed CHAR or VARCHAR columns with compatible collations. A WHERE last_name LIKE 'Sm%' condition on a plain index INDEX(last_name) typically turns into an index range scan rather than a full read of the table. Case sensitivity plays a part here. Columns stored with case insensitive collations can match Sm% for both Smith and smith, while case sensitive collations treat those as separate values.

Queries that blend equality and prefix search across multiple columns can still benefit from B tree indexes when the index column order aligns with the filter. Let’s say we are working with a support tool that keeps customer records with status flags and names:

Searches against active customers whose last names start with Sm fit this index:

The index is sorted first by status, then by last_name. With status = 'active' fixed and last_name LIKE 'Sm%' expressing a prefix, the engine can walk a compact section of the index that covers exactly those rows instead of scanning inactive records.

Case insensitive prefix search benefits from functional indexes in engines that support them. PostgreSQL, for example, can store lowercased names directly in the index:

Queries then match against the transformed value:

This query form avoids lowercasing every row during the scan and lets the engine use the index entries that already hold LOWER(last_name) values. Similar ideas appear in Oracle and in other systems that support indexes on expressions, with syntax differences but the same basic goal of lining up the index with the expression in the filter.

Prefix search with B trees works best when queries avoid wildcards at the front of the string and limit function calls on the indexed column. As long as filters keep that prefix intact and match the expression stored in the index, the engine has a good chance of treating the request as a range scan instead of a full scan.

Full Text Search For Word Based Matching

Applications that deal with article bodies, support tickets, product descriptions, or chat transcripts need more than prefixes. Users expect to type a few words and see matching content with some ranking applied. B tree indexes on plain text columns do not have enough structure for that style of search, so engines rely on full text indexes that act more like term dictionaries. Full text indexes usually work by breaking text into tokens and building an inverted index. Instead of mapping each row to its values, the system maps each term to the list of rows that contain it, sometimes with extra details such as term frequency or positions. This layout mirrors what search engines use and makes it possible to fetch candidate rows quickly for a given set of terms.

PostgreSQL supports full text search through tsvector and tsquery types combined with GIN or GiST indexes. A typical table can look like this:

The index stores the tsvector representation of title || ' ' || body. That representation contains normalized tokens for words in the text, usually lowercased and stripped of suffixes according to the selected configuration such as english. Queries build a tsquery value and apply the @@ operator to match those tokens:

In this code, the planner uses the GIN index to find rows whose token sets satisfy the tsquery. Binary operators in the query string such as &, |, and ! express logical conditions between terms, and weights or ranking functions can adjust ordering in result sets.

MySQL follows a similar concept with FULLTEXT indexes on CHAR, VARCHAR, and TEXT columns. Developers can declare an index like this on an InnoDB table:

Queries then use MATCH and AGAINST:

The FULLTEXT index organizes terms and document references so that searches over large text columns can avoid row by row scans. Boolean mode queries add operators such as + and - to require or exclude terms.

SQLite handles full text search through FTS virtual tables like FTS5. Rather than adding an index to an existing table, the database stores content inside a special table that maintains its own full text index:

Inserts and updates into notes_fts update the full text index automatically. Queries use a MATCH operator with a search string:

That search looks for rows where partial and search occur within two tokens of each other. Ranking functions such as bm25 can then order results by relevance.

All of these systems trade more complex index maintenance at write time for fast reads on multi word queries. Tokenization, stopword lists, stemming, and language configurations vary between engines, but the shared goal is to treat text as a bag of terms and store those terms in a structure that makes term based search fast.

Trigram Indexes For Fuzzy Partial Search

Substring search and typo tolerance call for a different strategy. Queries that look for fragments buried inside a word or short string, or that need some tolerance for misspellings, do not map neatly onto plain full text indexes. Many engines address that niche with trigram techniques and related extensions.

Trigram methods break strings into overlapping chunks of three characters. For the text search, one common set of trigrams would be sea, ear, arc, rch, sometimes with extra boundary markers added at the start and end. When an index stores these trigrams, the engine can compare trigram sets between the search term and stored values, then treat higher overlap as a sign of similarity or as a way to filter candidates for a substring match.

PostgreSQL brings trigram support through the pg_trgm extension. After enabling the extension, indexes can store trigram data and speed up LIKE, ILIKE, regular expression matches, and similarity operations.

With that GIN index on name, substring search across a large product catalog becomes more practical:

The planner can rely on trigram overlap between graph and stored names to pick a reduced set of candidates. Only those rows whose trigram sets share enough three character chunks with graph need to pass through the full ILIKE comparison. This narrows the search space even though the wildcard sits at the beginning of the LIKE pattern.

The pg_trgm extension also supplies similarity operators and functions, which support nearest neighbor style queries. One search for names close to a user supplied term uses one of these operators:

Typos such as elastc instead of elastic still return reasonable matches, ordered by their trigram similarity scores. A GIN or GiST index on the trigram representation keeps such searches responsive on large tables.

Other ecosystems adopt related ideas. MySQL offers an ngram full text parser that splits text into fixed length fragments and feeds them into a full text index, which works well for languages without clear word boundaries and also helps with substring matches in some workloads. Third party extensions and libraries for SQLite and other databases implement trigram or ngram search in various ways, always with the same basic goal of storing short character fragments in a way that supports partial and fuzzy matching on large data sets.

Conclusion

In practice, partial text search performance comes down to how string operators interact with index structures. LIKE with anchored prefixes rides on B tree ranges, full text indexes route word based queries through token lists, and trigram indexes narrow substring and fuzzy matches by comparing short character sequences. Matched mechanics let databases scan far less data, keep latency low, and still support flexible search behavior as tables grow.

1. PostgreSQL Text Search Documentation

2. PostgreSQL pg_trgm Extension Reference

3. MySQL InnoDB Full Text Search Guide

4. MySQL Ngram Full Text Parser

5. SQLite FTS5 Extension Overview

6. SQL Server Full Text Search Overview

Share Alexander Obregon's Substack