This one gives you a specification and asks you to build it. Exercises 1 to 3 gave you a working system and asked you to predict how it fails. Writing the specification is the other half of the skill, and it is the half an agent is genuinely good at taking off your hands — once you have it.
There is a worked answer in this directory. That is the same bargain the other three exercises make: nothing is hidden, because the answer is not the point. Build your own first if you want the exercise; the probes below run against either, and the probes are the part that teaches you something.
Deck reference: Day 1 §6.2 Reference Data — Get It On Demand versus Get It In Advance (ECST), and Be Honest About the Trade. Day 2 picks this up again as FBP — Where Do Lookups Live?
model/catalogue.py is a dict[str, float]. It has been standing in for reference
data that belongs to somebody else — the catalogue service owns SKUs and prices, and your
order service needs them to price an order.
That is the most common integration in any system, and there are exactly two answers to it. You have effectively been doing the first one:
| Get it on demand | call the catalogue service when you need a price. Always current. You are now down when it is down, and slow when it is slow — which is GIZMO-SLOW, and you have already watched what it does to your queue |
| Get it in advance (ECST) | the catalogue service publishes its changes; you keep a local copy and read that. Never blocked, never slow — and always a little bit out of date |
Start from 03-streams. Copy it somewhere of your own, and add:
- A
PriceChangedevent and a topic for it. A SKU and a price, on the Kafka stream gateway insimple_eventing/that exercise 3 gave you, keyed so that events for one SKU stay in order. Think about whether it should carry the new price or just say that the price changed — §6.3 Domain or Delta Event and Summary or Snapshot, and the choice matters here more than it looks. Probe D is where it stops being a matter of taste. - A seeder that publishes prices onto that topic, and can change one on demand.
- A consumer that follows the topic and maintains the local copy — in its own process, and in
a store that outlives it. Not a thread inside the receiver, and not a map in memory:
SQLite, which is a file on your own disk. Two reasons, and both of them are probes.
A separate process is a thing you can
kill -9, which is Probe B. A store that survives the kill is what makes Probes C and D mean anything at all — a map in memory is empty after every restart, so "my copy is stale" and "my copy is gone" look identical and you can never see the difference that ECST is actually about. Cataloguereads the local copy instead of its dictionary.price_ofno longer throwsCatalogueUnavailableError, because there is nothing left to be unavailable.
SQLite is the cheapest durable store there is, and in Python it is cheaper than that:
sqlite3 is in the standard library. Nothing to install, nothing to add to a
requirements file, and the database is a file you can delete. No container, no port, nothing
else to go wrong on a laptop.
This is the sharpest thing in the exercise, and it is exercise 1's lesson arriving a second time wearing different clothes.
Catalogue now reads from a database. If model picks up the SQLite library to
do that, you have put a storage technology in your domain exactly the way exercise 1 had a
broker in it, and you will have undone the fix you made in the first half hour.
So: Catalogue depends on PriceStore, an interface model itself
declares — "given a SKU, what is the price, and do you have one at all?" local_copy/
implements it, and is the only place that names SQLite. receiver.py puts the two together,
because composition is the application's job and not the domain's.
ECST moves the dependency; it does not remove it. The seam that kept RabbitMQ out of the domain keeps SQLite out of it, unchanged, and that is the point worth carrying out of this exercise: a seam you built for one reason pays for itself against a problem you had not thought of yet.
And it is a check rather than a matter of opinion. check_domain.py already
walks model/ and fails if anything under it imports the broker's library. Add
sqlite3 to the list it rejects. Python has no compiler to do this for you, and a
rule nothing enforces is a rule you will break in the second hour of a take-home.
If it does, the lookup has leaked into your domain the way pika.spec.Basic.GetOk did in exercise 1 —
and it does not have to. The handler asks Catalogue for a price; Catalogue
decides where prices come from. That is what the seam was for, and the handler's line is the
same line it was in exercise 3.
Notice what did and did not move. The store is out of process in the sense that a different
process fills it — but reading it is a file read on the same disk, measured in microseconds, and
nothing waits on a network while a message sits unacked. Put a Redis or a shared database on the
other side of that interface instead and you are making a network call from inside the handler
again, which is GIZMO-SLOW wearing a different hat and the thing ECST was supposed to buy you
out of. The interface is what lets you be wrong about that later without touching the domain.
GIZMO-SLOW and FLAKY-1 do not survive this exercise. They were on-demand failures — a
lookup that hung, a lookup that was briefly unwell — and there is no longer a call to hang or to
be unwell. You did not fix them. You removed the thing that could fail, and bought a different
failure in its place, which is Probe B.
Your agent can write all four steps, and step 3 is the one it needs you to have decided first: a local copy is not a specification until somebody says where it lives and what happens to it when the process dies. What an agent cannot do is answer the questions below, and those are the exercise.
Two to three hours to build, and an hour on the probes, if you let the agent write the code and spend your own time on the questions. It is longer than any of the three you did in the room; that is what take-home buys. You are done when:
- a price change published by the seeder shows up in an order placed a moment later
- you have a number for publish-to-applied staleness, from Probe A, that you measured rather than estimated
- you can say what your system does when the price consumer is dead (Probe B) and when it has never run (Probe C), and whether the same thing should happen to an order that arrived too early as to one for a SKU that does not exist
- you have killed the price consumer inside its own write window (Probe D) and can say what it cost you
PlaceOrderHandleris untouched, andmodel/has no SQLite dependency
These need four terminals — the receiver, the price consumer, the stream consumer if you want to watch the events, and one to run the seeder and the sender from. That is one more than any previous exercise asked for, and it is because the price consumer is a real process now.
python price_consumer.py # terminal 1, leave it running
python receiver.py # terminal 2, leave it running
python price_seeder.py seed # terminal 3: publish a starting price for every SKU
python sender.py # terminal 3: place an order
1. The catalogue publishes a price change. I place an order one second later.
Which price does the order get? ______
2. How would I *measure* the answer to question 1 rather than guess it? ______
3. What is the worst case, and what makes it the worst case? ______
Then measure it. Put a timestamp on PriceChanged when you publish it, and print the
difference when your consumer applies it to the local copy. Publish-to-applied is your
staleness, and it is the whole trade.
python price_seeder.py set WIDGET-1 11.99 # terminal 3, watch terminal 1 print the gap
python sender.py # then place an order and see which price it got
Measure it to the moment the order is priced instead and you will get about a second — which is how long you waited before placing the order, not what the broker cost you. That is worth doing once, deliberately, to see the difference between the two numbers.
On the machine these were written on it is 10 to 60 ms once the seeder is warm — and the same
probe, on the same broker, reads over half a second in one of the five languages. That spread is
not the broker. It is the connection: set starts a new seeder every time, so every set pays to
connect again, and how long that takes is a property of the client library rather than of Kafka.
The first record after any start is the slow one, and with a one-shot publisher every record is
the first record. Your number will differ and that does not matter; having measured one, and
knowing which part of it is yours, does.
One reading will look absurd, and it is honest. Replay a record that has been sitting in the log for ten minutes — which is exactly what Probe D makes you do — and publish-to-applied comes out at ten minutes, because that is how long ago the catalogue said it. It is measuring staleness, not latency, and on a replay those are the same arithmetic and very different facts.
▎ Behind by one broker hop, which is the trade. You now know what one broker hop costs on your laptop. It is not what it costs in production, but you know how to find out.
Kill the price consumer — a real kill -9, not a Ctrl-C, because a clean shutdown is the
case nobody has a problem with. Leave the order pump running. Publish three price changes. Place
orders throughout.
kill -9 <the price consumer's PID> # it prints it on startup
python price_seeder.py set WIDGET-1 49.99
python sender.py
4. Do the orders succeed? YES / NO
5. Are they priced correctly? YES / NO
6. Does anything, anywhere, report a problem? ______
7. How long could this go on before someone noticed? ______
Measured: the orders succeed, they are priced at whatever the copy last heard, the receiver logs an ordinary success for every one of them, and no queue, no log and no broker metric moves. The only thing anywhere that knows is the consumer group's lag, in a process that is no longer running to report it.
▎ This is the failure mode ECST buys you, and it is the quiet one. On-demand lookup fails loudly: the call times out and your queue backs up, which you saw in exercise 1. A stale local copy keeps answering, confidently, with last week's prices. Decide which of those two failures you would rather have, and then say how you would detect the one you chose.
Start everything from clean — which now means deleting the database as well as the topic, and noticing that you had to:
../00-setup/reset.sh
rm -f prices.db
Then place an order before you start the price consumer at all, and a second one for a SKU that really does not exist after the copy has filled up. You need both, and in that order, because with an empty copy every SKU looks the same — which is rather the point:
python sender.py # WIDGET-1: fine, but we have no copy yet
# ...now start the price consumer and seed it, then...
python sender.py poison # NOPE-404: not a thing, and never will be
8. Are those two the same failure? Say what the difference is. ______
9. Where does each of them end up, and how long does each one take? ______
10. Which of the two is your pump wrong about, and what would you
change to fix it? ______
Measured: the two orders report different things — cannot price 'WIDGET-1': the local copy has no prices in it yet and 'NOPE-404' is not in the catalogue — and then exactly the same
thing happens to both. Four attempts, about five seconds apart, and dead.streams.PlaceOrder.
▎ The domain told the truth and the policy ignored it, and that split is worth more than either half. Catalogue can only say what is wrong; the pump decides what to do about it, and exercise 2's pump has exactly two answers — a body it cannot read is never retried, and everything else is retried three times and then dead-lettered. Neither answer is right for these two. The unknown SKU is as permanent as an unreadable body and should never have been retried at all; the empty copy is transient and twenty seconds is a strange budget for "has the price consumer started yet". You do not fix either of those in the domain.
Now the half a map in memory could not show you. Seed the prices, place an order, then stop every process — the consumer, the receiver, all of it — and start them again without seeding.
11. Does the order still get priced? YES / NO
12. Where did that price come from, and how old is it? ______
13. The copy can tell you its age. Why is that still not enough? ______
Measured: the receiver starts, says holding 2 prices -- newest change 2m old, and prices
the order. No consumer is running. No seeder has run.
Read that line again, because it is the whole probe. The receiver knows how old the copy is. It says so. And then it prices every order that arrives, for as long as it runs, without ever mentioning it again — so the difference between a copy two minutes old and one from March is one line of startup logging that scrolled off the top an hour ago.
▎ A durable copy adds a third state, and it is the dangerous one. "No copy", "a current copy" and "an old copy" are three different things, and only the first is obvious from the outside. Carrying the time on every row is what makes the third one knowable at all, and it is a schema decision you make in step 1, long before anybody asks for it — get it wrong and no amount of monitoring can recover it. Getting it right, as the answer here does, buys you the ability to notice. It does not buy you noticing.
Exercise 3's Probe A was the receiver: a Kafka write and a RabbitMQ ack, two brokers, no transaction. You watched it and agreed it was unfixable in place.
This one is yours, and it is inside the price consumer. It applies a record to the local copy,
and it commits its offset, and those are two writes with no transaction between them. The window
widens the gap so you can aim at it, exactly as DUAL_WRITE_WINDOW did:
PRICE_WRITE_WINDOW=15 python price_consumer.py # terminal 1
python price_seeder.py set WIDGET-1 77.77 # terminal 3
kill -9 <the price consumer's PID> # while the window is open
python price_consumer.py # start it again and watch
Give the consumer twenty seconds to join before you publish, or you will sit watching a
terminal that has nothing to say. That is the group-join delay from Two things to know before you
trust a number in ../00-setup/README.md, and it is the single most
common reason one of these probes looks broken when it is not.
14. The price was applied but the offset was not committed. What happens
on restart? ______
15. Is that a problem? Say why, in one sentence. ______
Measured: lag.sh shows the partition sitting behind the log end, the restarted consumer reads
the same record again, and the copy ends up with the price it already had. Nothing was harmed and
nothing had to be clever.
Now reverse the two lines in price_consumer.py — commit the offset first, then apply —
and run it again.
16. What is missing now, and when will you be offered it again? ______
17. Which of the two orderings would you ship? ______
Measured, and this is the one to sit with: the copy still holds the old price, the restarted
consumer reads nothing at all, and lag.sh reports zero lag on every partition. The consumer
is caught up. The group is healthy. Every dashboard you have is green, and a price change has been
destroyed. Nothing will ever offer it to you again.
▎ One ordering applies twice. The other loses the price for ever. There is no third place to put that line — you have now proved it twice, in two different processes, and the second time it was your code. Whether the duplicate is harmless is not luck: it is the Summary or Snapshot choice you made in step 1, paying for itself two probes later. Choose a delta event — "the price went up by 2.00" — and applying it twice is simply wrong, and you have to solve this properly.
Put the apply back before the commit before you go on, if you intend to keep the code.
Day 2 builds this again, in the Paper Flow exercise, as the Catalogue Maker — and asks where a lookup lives in a flow-based design. If you have done this, you will have already met the answer.
The dual write you just met in Probe D is the one worth arguing about, because the obvious candidate is not it. The price consumer writes its copy, the order handler writes an order — those are two processes and two stores, and nobody expects one transaction across them. The one that catches people is the one inside a single loop that looks like a single step.
And the fix is the one exercise 3 named. An Outbox will not help here, because there is no database the offset lives in — the offset is Kafka's. What does help is making the apply idempotent, which a snapshot price already is, or moving the offset into the same store as the copy so that one transaction covers both. That second answer is what a stream-processing library does for you, and it is worth knowing that is what you are buying when you adopt one.