The first version of Replylume worked, technically, the week after we finished it. It could look up an order in Stripe, check a configured date window, issue a refund, and write a note back to the Zendesk ticket. We tested it on a handful of synthetic tickets and it ran cleanly. We thought the hard part was done.
The hard part was not done. What followed was four months of learning the distance between "technically working" and "operationally deployable," and every lesson from that period shaped how the product works today. This is an account of the decisions that mattered, the ones we got wrong, and what we'd do differently if we were starting week one over.
Lesson 1: Configuration Is the Product
The first thing we discovered when we onboarded our first pilot team was that the configuration interface -- the settings where you define resolution criteria -- was more important than the resolution engine itself. Our engine worked. The criteria configuration was a JSON file that required editing by hand and redeploying the service.
Our first pilot customer's support ops lead spent six hours on the phone with Daniel trying to define their refund criteria precisely enough to produce the right behavior. Not because the criteria were complicated -- they had a 30-day window, a $150 threshold, and three excluded product categories -- but because the format for expressing those criteria was ours, not theirs, and the gap between "what the criteria are" and "how to express them in our system" was significant.
We built a configuration UI in the first two weeks after that experience. The JSON file never shipped to a second customer. The lesson wasn't just "build a UI" -- it was that the criteria definition process is part of the product's core value proposition. A team that can clearly articulate their refund policy in our configuration interface has, in the process, documented their policy more explicitly than they likely had before. That documentation is useful to them independently of whether automation runs on top of it. We should have recognized that earlier and built for it.
Lesson 2: The Escalation Path Matters as Much as the Resolution Path
The first version of Replylume had detailed thinking about what happened when a ticket met resolution criteria. It had almost no thinking about what happened when a ticket didn't. The escalation flow was: criteria not met, route to agent queue, add a tag. No context note. No reason code. No indication of what the automation had checked before escalating.
Agents in the pilot described the escalated tickets as "black boxes" -- they arrived in the human queue and required agents to do all the context assembly from scratch, with no indication of why the automation had declined to resolve them. The escalation was often the first time an agent looked at the ticket, and they had no information advantage over what they would have had if automation hadn't touched the ticket at all.
The context note system we built later -- specifying the escalation reason, what data was checked, the recommended action type -- reduced agent time on escalated tickets by about 35% in the cohort data. The engineering work to build it was two weeks. The delay in building it was a choice we made because we prioritized the happy-path resolution flow over the exception path. That was the wrong priority order. In practice, the exception path is where customer experience is most at risk, and the quality of the handoff to a human agent is what determines whether the automation is a net positive or negative for the complex cases.
Lesson 3: Start with the Billing Integration, Not the Helpdesk Integration
Our initial integration architecture connected to the helpdesk first -- Zendesk webhook, ticket data, routing logic -- and then called the billing system as a secondary step during resolution. That order was wrong, and the mistake cost us two weeks of debugging subtle failures.
The billing system is the most brittle part of the stack. Stripe API responses have edge cases -- rate limiting, occasional timeouts on specific charge record types, retry behavior on refund submissions -- that are not well-documented and require empirical discovery. The helpdesk integration, by comparison, is well-documented and predictable. Building on top of the helpdesk while the billing integration was still unpredictable meant that failures in the billing layer surfaced as confusing behaviors in the helpdesk layer, making debugging significantly harder than if we'd validated the billing integration in isolation first.
If we were starting over, we'd build and validate the billing integration -- specifically the refund issuance path with full error handling and retry logic -- before connecting the helpdesk webhook at all. The resolution engine should be able to run from a structured input object (ticket type, order ID, criteria) without depending on a live helpdesk connection during development. We built that abstraction eventually; we should have started with it.
Lesson 4: Audit Log from Day One
The audit log -- the record of every automated action, with timestamp, decision inputs, and outcome -- was on our roadmap but not in the first version. We planned to add it after we validated that the core resolution flow worked. Two weeks into the pilot, our first customer asked for a log of what Replylume had done in the previous 24 hours. We didn't have one.
The absence of the audit log was the only thing that came close to stalling the pilot. Not because there were errors the customer needed to investigate -- there weren't -- but because the trust model for automated billing actions requires verifiability. "This refund was issued automatically, and here's the record" is a categorically different conversation from "this refund was issued automatically, trust us." Both outcomes may be identical, but the first one is auditable and the second one is not. The customer's security and finance teams were waiting on the audit log before they would expand the pilot scope.
We shipped a basic audit log in a weekend sprint. The lesson is that audit logging isn't a nice-to-have feature for a system that takes automated billing actions -- it's a prerequisite for the trust model that the system requires to function in a real organization. It should have been day one, not week four.
Lesson 5: The Criteria Design Conversation Is a Product Feature
The most valuable thing we did with our first pilot customers wasn't shipping code. It was sitting on calls while they defined their resolution criteria, asking questions, and observing where the definition process got hard. Every place where a support ops lead said "well, it depends..." or "usually, but not if..." was a signal that the criteria-definition interface needed to handle that type of ambiguity explicitly, rather than forcing it into a binary rule that would produce wrong outputs at the boundary.
That observation directly shaped the configuration UI. The "it depends on customer history" case became the context-based escalation trigger system. The "usually, but not for enterprise accounts" case became the product category exclude-list. The "we'd normally approve this but the customer has been abusive" case became the manual-review-always flag at the customer account level. Each of these features is a direct encoding of a "it depends" from a real pilot customer's criteria design session.
The configuration interface is not just a way to enter settings. It's a structured conversation about what your support policy actually says, made explicit. We could have built a more powerful configuration system faster if we'd started earlier with the observation that the configuration conversation itself is where the most valuable information about the product's requirements lives.
What We'd Do in Week One
Knowing what we know now, week one would look different. We'd start by building the billing integration in isolation, with full error handling and a test harness that could simulate the full range of Stripe response types. We'd add the audit log to the data model before anything else. We'd sketch the configuration UI before writing a single line of resolution logic, because the UI is where the requirements live. And we'd get one real support ops manager on a call in week two -- not to demo anything, but to watch them define their refund criteria out loud, and learn from every place where the definition got complicated.
The resolution engine was the part we thought was the product. The configuration, the audit trail, the escalation handoff quality, and the trust model for automated billing actions are the product. The engine is just what runs on top of all of that. We got there eventually; we should have gotten there first.