Delivery
A DevOps interview on shipping a change in order, with a flag, a migration, and a rollback that still exists.
These prompts are original practice. They are not questions from a named employer or a live hiring loop.
1.A payroll release renames employee_id to worker_id and deploys the new app in the same step. Pods still on the old image insert into employee_id, which is already gone. What should have shipped before any drop?
2.The lunch-payment app adds NOT NULL on seat_label while older instances still insert rows that leave it empty. The migration fails, and new pods then crash on insert. Which order avoids that?
3.Pharmacy refills gain the status recalled. The new app writes it. Old pods still serving lookups throw when they see a value outside their list. What must be true before the first recalled write?
4.Half the warehouse API pods are new. New handhelds send bin_code. Old API pods reject any JSON field they do not know and drop the scan. What should have deployed before handhelds sent bin_code?
5.The ferry fare flag service times out. Each booking pod treats a timeout as flag on and shows the new fare screen you were still canarying. What behavior is missing on a timeout?
6.You turn the ferry flag off after bad fares. Each pod read the flag once at process start, so the old fare returns only after a restart, and a restart means another deploy. What should the pod do?
7.A canary marks 5 percent of map-tile pods. The load balancer sends those pods only the static health URL. Tile errors on the other pods never move. What did this canary measure?
8.The clinic portal's readiness probe fetches index.html from an nginx sidecar. The app container is crash-looping, nginx still returns the file, and the rollout continues. What can that green probe claim?
9.You put the previous billing image back after a bad release. The migration has already replaced amount_text with amount_cents and rewritten every row. The previous image only reads amount_text and crashes. What did the image rollback leave in place?
10.Production is supposed to run the image CI built from commit 9c1. A laptop retags latest onto a local build, and the cluster pulls latest. What is production running?
11.A database password sat in a Helm values file. The next commit deletes the file. The password is still valid on the database. What is still true about the secret?
12.A rolling update has replaced 30 of 90 pods. The autoscaler then adds 10 pods from the old template. Users still hit the bug you are replacing. What is unfinished?
13.A chat migration drops messages.body in the same change that adds body_v2. A budget leaves one in ten pods on the old image, and those pods only read body. What fails?
14.A ranking canary shares one error graph with the stable fleet. The stable fleet is large, so bad canary queries barely move the line, and the pipeline promotes. What did the shared graph hide?
15.The rollback script runs the migration's down step, which drops amount_cents. Bills written after the up step exist only in that column. What does running down destroy?
16.Two builds of commit 9c1 produce two digests. The Dockerfile installs whatever the mirror currently calls the latest patch of a system package. What changed between the builds?
17.Someone deletes config.env in a follow-up commit and says the token is gone. A release branch still builds from the earlier commit. Where is the token?
18.A new order message adds tax_breakdown. You deploy the producer first. Old consumers crash on the field and the queue backs up. Which order keeps orders moving?
19.The new checkout is a flag compiled into the phone binary from the app store. You set a server flag to off. Phones that already updated still show the new checkout. Why does the server change do nothing?
20.The app rollout runs a migration that builds an index by locking the orders table. The lock waits behind a report, and the fleet stays half upgraded for forty minutes. How should that index get built?
21.Green and blue share one ticket database. Green's migration rewrites seat rows into a shape blue cannot read. The router can still send everyone back to blue in one command. What does that flip do?
22.Monday's fare canary is compared with last Sunday's error rate. Sunday had a tenth of the traffic and no commute rush. The canary looks worse and an automatic rollback removes a good fix. What is wrong with the baseline?
23.The new tax path is on for staff accounts only. Staff orders are tax-exempt, the trial looks clean, and you enable the path for guests. Guests then get wrong totals. What did the staff trial skip?
24.A rolling restart sends SIGTERM and the ticket process exits immediately. The pod is already out of the load balancer, and in-flight purchases on open connections die. What window is missing?
25.The rollout sets maxSurge to 100 percent, so the pod count doubles. Each pod opens a full database pool. Purchases fail with connection errors even though the new code is fine. What did the surge use up?
26.Each new pod runs the schema migration as an init container. Forty pods start together and forty sessions lock the same table. What should run that migration?
27.The pipeline applies a new ConfigMap and reports the cluster in sync. Pods keep the tax rate they loaded at start. Customers see mixed rates only after an unrelated restart. What did the sync not update?
28.You need the previous digest after a bad release. Registry cleanup has deleted every image except the one currently deployed. The rollback command has a name and nothing to pull. What should cleanup have kept?
29.Signature checks on the production cluster are off so a person can move a tag by hand. Anyone who can push that tag can run code in the cluster. What should admission require?
30.An attestation names digest abc from commit 9c1. Production runs digest def under the same tag, and the ticket attaches the abc file anyway. What is the gap?
31.A hotfix is edited onto the live deployment with kubectl. An hour later GitOps sync restores the git manifest, which lacks the fix, and the bug returns. Where should that hotfix have landed?
32.Helm rollback restores the previous deployment and replays a migration hook. That hook's down step drops a column written all day. What do you need to know before using that rollback?
33.Every receipt is written to amount_text and amount_cents. Errors on the cents write are logged and ignored. Reads still use the text column, so receipts look fine. What does a later read cut find?
34.Receipt reads switch to amount_cents while a backfill of old rows is still running. Old receipts show a blank amount. The text column still has the number. What should have finished first?
35.A parity job compares one receipt, sees matching cents, and you switch every read. Refunded receipts are wrong later. The job never opened a refund. What was too small?
36.The checkout flag is on for 10 percent by drawing a random number on every request. The same passenger sees the new fare, then the old fare, on two refreshes, and the booking splits. What should the draw use?
37.New pricing pods cache a fare quote under a key the old pods also use. Old pods do not know the tax field and write the key back without it. A passenger who hits both versions loses the tax. How should the new quote be stored?
38.The load balancer caches a successful health response for five minutes. The pod starts failing purchases right after that cached 200. Traffic continues until the cache entry expires. What is the cache concealing?
39.Liveness runs a query on the primary database. A one-minute stall fails the probe on every pod. The cluster restarts the fleet, everything reconnects at once, and the stall gets worse. What should liveness decide?
40.You restore the previous image, and the remote fare flag stays on. That image treats an unknown flag as on and keeps running its older broken fare path. What handling turns the rollback into a real stop?
41.A contract job drops employee_id the same afternoon the new code deploys. A disruption budget keeps three old pods, and those pods still select employee_id. What ran too early?
42.A disruption budget stops the rollout at 70 percent. The pipeline timer expires and the job is marked successful. Thirty percent of pods still run the old digest and the old bug. What is a real success signal?
43.One pipeline step migrates the ticket table and shifts traffic onto the new image. The migration fails halfway. New pods are already selling against a half-built schema. How should those steps be split?
44.The release workflow checks out the moving tag rel-2026 and builds it. After review, someone points rel-2026 at another commit. The attestation still records the tag name. What should the build have checked out?
45.The secret scanner reads the pull request as plain text. A live token committed as base64 in a YAML field does not match the scanner's prefix list. What reached the branch?
46.City A's edge proxies speak a new handshake. City B's proxies are still on the old one. The cities call each other for ferry inventory, and the new handshake drops the old one. What has to hold during the partial roll?
47.The mesh sends 10 percent of user traffic to the canary. A nightly settlement job uses pod IPs it cached yesterday and only reaches old pods. You judge the canary by that job's success. What did you actually see?
48.The search mapping now requires the field aisle. Old indexers still bulk-write documents without it. The bulk call fails and the index queue grows. What should the mapping allow until every indexer is new?
49.You roll the search indexers back to the previous image. They refuse to start because they cannot parse the new mapping, which stayed in the cluster. What did the image rollback leave behind?
50.Fare flags are stored in the same database the migration is locking. Flag reads stall, and you cannot turn the new fare off while the lock is held. What did this design tie together?
51.The canary handles one request, the probe returns 200, and the pipeline shifts the rest of the fleet. The next thousand requests show the bug. What was skipped?
52.An automatic rollback restores the old image while a backfill rewrites fares into the new shape. The old image crashes on a rewritten row. What should have been paused with the rollback?
53.Friday's train ships five changes in one image. One corrupts fare quotes. Rolling the image back also removes four fixes you still want. What would have kept those four in production?
54.The status line says rolled back after the previous image is restored. The fare migration is still applied and is still rewriting quotes. What should the status say?
55.Some nodes pulled app:1.4 yesterday. You move the tag 1.4 to a new build today. Nodes that pull now get different bytes. The deployment name never changed. What are users hitting?
56.One shared runner builds untrusted pull requests and also signs release images. The signing key is on that runner. A malicious script in a pull request prints the key. What must an untrusted build be unable to do?
57.A Dockerfile sets the database password in an image layer. The line is removed and a new image is pushed. The previous image remains in the registry, and nobody rotates the password. Who can still use it?
58.A failed deploy prints the process environment, including the release signing key, into the CI log. Every engineer can read that log. What happens to the key?
59.Sticky sessions keep a passenger on one pod. Every pod is new except those with live sticky sessions, which still write the old fare column. A contract job then drops that column. What do those sessions hit?
60.During a rollout, CPU on the old pods spikes and the autoscaler adds pods to the old set. Each added pod serves the bug you are leaving. How should scaling behave while both versions exist?
61.The site starts serving HTML that points at /static/fare-9c1.js. That file is not in the bucket yet. Passengers get a page whose fare script fails to load. What should have been published first?
62.You turn on a flag that swaps in HTML referencing a chunk you forgot to upload. Staff on the flag see a broken page. Everyone else is fine. What has to exist before the flag moves?
63.You restore HTML that names the old assets, then delete the new chunk. Browsers and the CDN still hold the new HTML for its cache lifetime, and that HTML requests the chunk you deleted. What is part of this rollback?
64.A migration rewrites the huge orders table to fill a default, and it does this during the lunch rush. Writes queue behind the rewrite. The new column is not read yet. What is a calmer way to add it?
65.The migration statement timeout is 30 seconds. The orders index takes two minutes, so the statement is killed and leaves an invalid index. The pipeline retries and starts a second build on the same name. What was wrong with that timeout?
66.A check on fare_code is added in the form that validates every existing row under a strong lock. Ticket sales stall. The existing rows are already clean. What two-step add avoids that stall?
67.A migration adds a trigger that writes an audit row on every ticket update. Old pods, which were not part of this feature, now hold the ticket lock while the audit insert runs. Sales slow on the old code. Who does the trigger affect?
68.A migration recreates the ticket view and puts a new column first. An old report uses SELECT * and treats the first column as the ticket id. It now reads a label and files the wrong id. What should have been stable?
69.A migration creates an archive table and, in the same step, revokes the old app user's access to tickets. Old pods are still serving and suddenly cannot read tickets. What has to remain until those pods are gone?
70.You change the database to the new password, then start a rolling restart so pods pick up the new secret. Pods not yet restarted still have the old password and cannot log in. What should the database have accepted during that window?
71.The database password arrives as an environment variable. You remove the old password as soon as the secret object changes. Processes keep the old environment until restart, and the restart has not begun. What went too early?
72.The gateway sends half of traffic to API v2. V2 requires a header the current phone app does not send, so those calls fail. The other half still hits v1 and works. What must be true before the weight moves?
73.The canary is one pod out of two hundred. A noisy neighbor on its node makes that pod slow, and the pipeline rolls back a change that is fine on a quiet node. What is weak about this sample?
74.You canary overnight. The new path gets almost no purchases, the error count stays at zero, and the pipeline promotes at 7 a.m. The morning rush hits the bug. What did zero errors mean overnight?
75.The write flag stores fares only in the new column. The read flag is still off, so the booking page reads the old column and shows a blank after a successful save. What should have been true before relying on that writer?
76.You turn the read flag on first. The page reads amount_cents, writers are not filling it yet, and every fare is blank. The old column still has the numbers. What should have started first?
77.The rollback plan is to revert the git commit. That commit holds the app change and a migration that has already run in production. After the revert, the previous app cannot read the new column. What did the git revert leave untouched?
78.A job sees a down migration appear because the commit was reverted, and it runs that down step in production. Amounts written since the morning live in the column down drops. What should decide whether down runs?
79.Staging tested digest abc from commit 9c1. Production rebuilds from the branch tip, which has moved two commits, and deploys that new digest. You describe it as the build staging tested. What did production deploy?
80.Staging and production run the same image digest. Production's config turns the fare flag on. Staging left it off, and the staging run was clean. What did that run not cover?
81.A pull request from a fork runs in the release workflow, where the signing key is available. Its script signs an image it built and pushes the signature. What should a fork job not receive?
82.The image signing key lives in a private folder of the repo so laptops can sign hotfixes. A contractor laptop is stolen. The passphrase was in the same folder. What can that laptop do until you rotate the key?
83.Every web pod is on the new image, which only reads body_v2. A scheduled worker in another cluster is still old and still writes body. New web pods show empty messages for those chats. What did the web rollout miss?
84.When the database is down, the app returns HTTP 200 and the body unavailable, so the readiness probe stays green. The rollout replaces the fleet with pods that cannot sell tickets. What should that dependency failure do to readiness?
85.Readiness requests /health, and /health skips the auth middleware. This release breaks auth. The probe stays green and every login fails. What did the green probe fail to cover?
86.The canary's CPU falls, so the pipeline promotes. CPU fell because the new code rejects the purchase in a millisecond instead of doing the work. Purchase errors are up. What should have blocked promotion?
87.You change fare_code from text to an integer. Old pods still write N/A for an unknown fare. The type change aborts, and a retry locks the table during sales. What has to happen before that type change?
88.A migration adds a foreign key from the hot ticket row to a new coupon table. Every ticket insert now checks that table and contends with coupon writes. Ticket sales were not supposed to change yet. What arrived too early?
89.The in-code default for the new fare flag is true. A pod that cannot reach the flag service serves the new fare to everyone. That path was supposed to be a canary. What should the in-code default be?
90.The image ships a flags file with the fare flag on. If the remote override fails to load, the pod uses the file, and a remote outage enables the new fare for the whole fleet. What is the safe failure?
91.A security patch and an unfinished fare experiment share one image. You cannot ship the patch without also running the experiment, and the experiment is not ready. The hole stays open. What should the patch have been?
92.The security patch sits on the fare-flag branch, and the flag defaults to off. Production keeps running the old fare and never loads the patched branch. A scanner still reports the hole. What is wrong with that placement?
93.A developer runs today's migration from a laptop against production because the pipeline is slow. It succeeds. Nothing records the file, the person, or which database it hit. The next pipeline run starts the same migration. What is missing?
94.Two pipelines see the same pending migration and start it together. They deadlock on the ticket table, both mark the step failed, and a retry starts a third copy. What should guard the runner?
95.The runner inserts migration 18 applied and autocommits that row, then starts the schema change. The process dies before the schema change commits. The next start sees the row and skips 18. The column is missing. What was recorded too soon?
96.The canary points at a fresh database copy with no ticket rows. Its migration finishes instantly. On production the same statement locks a huge table. You treat the canary as a test of the migration. What did the empty copy never show?
97.Five percent of pods are in the canary. The migration in that release rewrites every fare row before the first canary pod starts. Rolling those pods back does not restore the rows. What is the migration's blast radius?
98.The cluster pulls app:release from a public mirror. Someone replaces that tag with a different image. Nothing checks a signature. What should the pull have required?
99.During an incident someone edits a file inside one running container. Purchases succeed on that pod only. A later rolling restart deletes the edit, and until then the fleet disagrees with itself. What is that edit?
100.A token never appears in a file. It is written in a merge commit message. The scanner reads blobs in the tree and reports a clean history. What did the scan miss?
101.You roll a new tax rate a pod at a time. For twenty minutes two pods charge two different rates for the same ticket, and finance cannot explain one price. When is a slow roll a poor fit?
102.You delete the fare flag while old images that still read it are running. Those images take the in-code default, which is the new fare, and the canary becomes the whole fleet. What should deletion wait for?
103.New pods log a purchase in a shape the log pipeline rejects. Lag looks quiet because the pipeline drops the unknown shape. You promote, and the purchase logs you would need in an incident are gone. What should have been ready first?
104.The deploy job marks the release healthy because its call to /health returned 200. It never places a ticket. After promotion, ticket sales fail a validation that /health does not run. What should the job have exercised?
105.You expand the schema, start the new image, and schedule a contract job to drop the old column after one hour. The rollout sticks at 80 percent for two hours because a node is bad. The timer still fires. What is wrong with the timer?
106.The previous digest is still in the registry. The deployment records only the tag app:current, which you have since moved. You can no longer tell which digest a rollback should name. What should the release record have stored?
107.A migration adds a column the new code requires. You deploy the code first. New pods crash-loop because the column is absent, and the crash loop makes the later migration's lock slower. What order avoids the crash loop?
108.You dual-wrote fares for a week, checked parity, and dropped the old column. A back-office tool in another repo still reads that column and now fails every refund. What inventory was short?