Agent Needs Walkthrough

How this application addresses each stakeholder requirement from the discovery sessions

Sarah Chen, Deputy Director of Label Compliance

Need: Fast Performance (< 5 seconds)

“The system would take 30, 40 seconds sometimes to process a single label. Our agents just went back to doing it by eye because they could do five labels in the time it took the machine to do one. If we can’t get results back in about 5 seconds, nobody’s going to use it.”

Solution Implemented:

Browser OCR: ~2.5s average (162 image benchmark)
Claude Vision: ~3-4s via OpenRouter API
Smart Edge-Strip Rotation: 51% faster than full-image rotation
Confidence-Gated Retry: Only retries low-quality images when fast

Result: All processing completes in under 5 seconds, meeting the critical performance requirement.

Need: Batch Upload Support

“During peak season, we get these big importers who dump 200, 300 label applications on us at once. Right now we literally have to process them one at a time. If there was some way to handle batch uploads, that would be huge.”

Solution Implemented:

Multi-file drag-and-drop: Upload 200+ labels at once
Queue processing: Progress tracking for each label
CSV export: Download all results for record-keeping
Review Queue: Centralized dashboard for batch review

Result: Agents can process large batches efficiently without manual one-by-one uploads.

Need: Extreme Ease of Use

“We need something my mother could figure out—she’s 73 and just learned to video call her grandkids last year, if that gives you a benchmark. Half our team is over 50. Clean, obvious, no hunting for buttons.”

Solution Implemented:

Single-page workflow: No complex navigation required
Guided 8-step walkthrough: Interactive tutorial panel
Visual feedback: Clear progress indicators at every step
Large, obvious buttons: No hunting for controls
TTB.gov visual identity: Familiar government website design

Result: Interface designed for users with varying tech comfort levels.

Marcus Williams, IT Systems Administrator

Need: Standalone Prototype (No COLA Integration)

“For this prototype, we’re not looking to integrate with COLA directly—that’s a whole different beast with its own authorization requirements. Think of this as a standalone proof-of-concept that could potentially inform future procurement decisions.”

Solution Implemented:

Self-contained deployment: No COLA system dependencies
Independent testing: Can be evaluated without integration
Mock data: 115 real TTB COLA records for realistic testing
RESTful API: Future integration path documented

Result: Fully functional standalone prototype that demonstrates value without system integration.

Need: Firewall-Friendly Architecture

“Our network blocks outbound traffic to a lot of domains, so keep that in mind if you’re thinking about cloud APIs. During the scanning vendor pilot, half their features didn’t work because our firewall blocked connections to their ML endpoints.”

Solution Implemented:

Dual OCR approach: Browser-based Tesseract.js works offline
Graceful degradation: Claude vision is optional enhancement
No hard dependencies: Core functionality works without external APIs
Configurable: Can disable cloud APIs via environment variables

Result: Works in restricted network environments with firewall constraints.

Dave Morrison, Senior Compliance Agent (28 years)

Need: Intelligent Fuzzy Matching

“The thing about label review is there’s nuance. You can’t just pattern match everything. Like, I had one last week where the brand name was ‘STONE’S THROW’ on the label but ‘Stone’s Throw’ in the application. Technically a mismatch? Sure. But it’s obviously the same thing. You need judgment.”

Solution Implemented:

Levenshtein distance: Measures similarity between strings
Case-insensitive matching: “STONE’S THROW” = “Stone’s Throw”
Punctuation normalization: Handles apostrophes, hyphens, spaces
Substring matching: Finds partial matches in longer text
Configurable thresholds: Balance precision vs recall
Human judgment preserved: Agent makes final decision

Result: System handles real-world variations while keeping human oversight.

Jenny Park, Junior Compliance Agent (8 months)

Need: Exact Warning Statement Validation

“The warning statement check is actually trickier than it sounds. It has to be exact. Like, word-for-word, and the ‘GOVERNMENT WARNING:’ part has to be in all caps and bold. I caught one last month where they used ‘Government Warning’ in title case instead of all caps. Rejected.”

Solution Implemented:

Pattern matching: Detects “GOVERNMENT WARNING:” in all caps
5 fallback strategies: Handles fragmented OCR text
Truncation logic: Stops at “health problems” end-marker
OCR error tolerance: Handles GOVERNMEN, GOVERNMENI, WARNIN6
76 unit tests: Comprehensive edge case coverage

Result: Strict validation with tolerance for OCR imperfections.

Need: Handle Poor Quality Images

“It would be amazing if the tool could handle images that aren’t perfectly shot. I’ve seen labels that are photographed at weird angles, or the lighting is bad, or there’s glare on the bottle. Right now if an agent can’t read the label they just reject it and ask for a better image.”

Solution Implemented:

Perspective correction: Corner drag + mesh warp for angled labels
Cylindrical unwrap: Handles curved bottle labels
Smart edge-strip rotation: Detects and rotates 90° warnings
Sharpen filter: Laplacian unsharp mask for blurry images
Confidence-gated binarization: Retries with Otsu threshold if needed
Auto-flatten: Removes perspective distortion automatically

Result: Comprehensive image preprocessing reduces rejections due to image quality.

Summary: All Stakeholder Needs Addressed

Performance & Usability

✅ <5 second processing time
✅ Batch upload (200+ labels)
✅ Intuitive interface for all skill levels
✅ Guided walkthrough tutorial

Technical Requirements

✅ Standalone prototype (no COLA integration)
✅ Firewall-friendly (offline OCR option)
✅ Fuzzy matching for real-world variations
✅ Human judgment preserved

Validation Accuracy

✅ Exact warning statement validation
✅ OCR error tolerance (76 test cases)
✅ Category-aware rules (beer/wine/spirits)
✅ 153 unit tests for reliability

Image Processing

✅ Perspective correction
✅ Cylindrical unwrap for bottles
✅ Smart rotation for edge warnings
✅ Sharpen & binarization filters