[{"data":1,"prerenderedAt":587},["ShallowReactive",2],{"case-study-trustclarity-observability":3,"related-case-studies-trustclarity-observability":186},{"id":4,"title":5,"body":6,"date":159,"description":12,"extension":160,"featured":161,"image":18,"imageAlt":17,"impact":13,"meta":162,"navigation":163,"path":164,"rotation":165,"seo":166,"slug":148,"stem":180,"technologies":181,"__hash__":185},"caseStudies/case-studies/trustclarity-observability.md","TrustClarity: OpenTelemetry and SigNoz Observability on AWS ECS",{"type":7,"value":8,"toc":149},"minimark",[9,14,19,23,28,34,38,42,47,52,57,61,65,68,71,106,109,113,120,135,144],[10,11],"case-study-header",{"description":12,"impact":13,"title":5},"Deployed OpenTelemetry and SigNoz on AWS ECS for a B2B surplus equipment platform — instrumenting 5 production services with 22 Terraform-managed alerts, then hardening ClickHouse after a silent data-loss incident.","5 production services instrumented with OpenTelemetry; 22 Terraform-managed alerts to Microsoft Teams; ClickHouse disk-full silent data loss diagnosed and hardened",[15,16],"case-study-image",{"alt":17,"src":18},"Manga-style illustration of glowing telemetry traces and dashboard gauges as energy streams","/case-study-media/tc-observability.jpg",[20,21],"impact-metrics",{":metrics":22},"[{\"value\":\"5\",\"label\":\"Services\"},{\"value\":\"22\",\"label\":\"Alerts\"},{\"value\":\"OTel\",\"label\":\"Instrumentation\"}]",[24,25],"project-links",{"project-label":26,"project-url":27},"VIEW SITE","https://trustclarity.com",[29,30],"project-overview",{"role":31,"team":32,"timeline":33},"Solo Full-Stack Consultant — observability architecture, OpenTelemetry instrumentation, Terraform alerting, and AWS ECS operations","Solo engagement","Mar 2026 — SigNoz stack, Phase 1 instrumentation, knowledge-graph two-tier alerts, ClickHouse hardening",[35,36],"challenge-section",{"challenge":37},"TrustClarity ran five active production services with zero observability. Latency regressions, dependency failures, and telemetry pipeline outages could not be detected until users reported them. The platform needed a self-hosted OpenTelemetry + SigNoz stack on AWS ECS, consistent instrumentation across Node, Next.js, Express/GraphQL, FastAPI, and a legacy React 16 app, and Terraform-managed alerts that would reach the team in Microsoft Teams.",[39,40],"results-section",{":results":41},"[\"Deployed SigNoz on AWS ECS (ClickHouse, ZooKeeper, OTel Collector, SigNoz UI) with VPC networking, EFS persistence, ALB, and Terraform-managed infrastructure\",\"Instrumented 5 production services with OpenTelemetry — server traces, browser Web Vitals, log correlation, and custom knowledge-graph metrics\",\"22 Terraform-managed alerting rules across services plus ClickHouse disk usage, all routing to Microsoft Teams\",\"Two-tier alert model for the knowledge-graph / GraphRAG service — trace-based service-health alerts and custom-metric alerts for graph queries, LLM calls, and embeddings\",\"Diagnosed silent OTel data loss when ClickHouse's internal query log filled a 20 GiB EBS volume; live-expanded to 50 GiB, added TTL retention, and shipped an 80% disk alert\"]",[43,44],"solution-section",{":technologies":45,"solution":46},"[\"OpenTelemetry\",\"SigNoz\",\"AWS ECS\",\"Terraform\",\"ClickHouse\",\"Node.js\",\"Next.js\",\"FastAPI\"]","Designed and operated a self-hosted SigNoz observability stack on AWS ECS, then instrumented TrustClarity's five production services with OpenTelemetry. Alerts are defined in Terraform and notify Microsoft Teams. After a silent ClickHouse disk-full incident dropped all telemetry writes, the stack was hardened with TTL retention and a hostmetrics-based disk-usage alert.",[48,49,51],"h2",{"id":50},"project-details","Project Details",[53,54,56],"h3",{"id":55},"what-problem-did-zero-observability-create-across-five-production-services","What problem did zero observability create across five production services?",[58,59,60],"p",{},"TrustClarity had five active production services and no shared view of latency, errors, or dependency health. Failures in the corporate site, buyer app, admin API, knowledge graph, and legacy frontend were invisible until someone noticed broken behavior. Without traces and alerts, there was no way to tell a slow graph query from a client-side regression, or to know whether the telemetry pipeline itself had stopped accepting data.",[53,62,64],{"id":63},"how-was-opentelemetry-and-signoz-stood-up-on-aws-ecs","How was OpenTelemetry and SigNoz stood up on AWS ECS?",[58,66,67],{},"I designed and deployed SigNoz on Amazon ECS: ClickHouse for signal storage, ZooKeeper, the OpenTelemetry Collector, and the SigNoz UI. Persistence used EBS for ClickHouse and EFS for SigNoz state, with an ALB in front of the UI and OTLP ingest. Infrastructure and alerting lived as code, with Microsoft Teams as the notification channel. The SigNoz UI stays private; browser telemetry from public visitors is proxied through the corporate site's Express server rather than exposing a public observability URL.",[58,69,70],{},"All five services were instrumented to report to SigNoz:",[72,73,74,82,88,94,100],"ul",{},[75,76,77,81],"li",{},[78,79,80],"strong",{},"Corporate site"," (Node.js/Express): server traces plus browser Web Vitals via a proxy endpoint",[75,83,84,87],{},[78,85,86],{},"Buyer app"," (Next.js): browser-side OpenTelemetry SDK",[75,89,90,93],{},[78,91,92],{},"Admin API"," (Express + GraphQL): server traces with structured-log correlation",[75,95,96,99],{},[78,97,98],{},"Knowledge graph"," (FastAPI): manual spans and custom metrics for graph queries, LLM calls, and embeddings",[75,101,102,105],{},[78,103,104],{},"Legacy frontend"," (React 16): pre-bundle pattern to work around an older Webpack toolchain",[58,107,108],{},"Per-service Terraform alerts cover P95 latency, error rates, and absence detection where appropriate. The knowledge-graph service uses a two-tier model: Tier 1 for service health (absent traces, error rate, route-specific latency) and Tier 2 for dependency lag on graph, LLM, and embedding backends — so slow dependencies can fire even when the request eventually succeeds.",[53,110,112],{"id":111},"what-did-the-clickhouse-incident-teach-and-what-shipped","What did the ClickHouse incident teach, and what shipped?",[58,114,115,116,119],{},"In March 2026, traces and metrics stopped appearing in SigNoz while the collector still returned HTTP 200. ClickHouse was rejecting writes because its ",[78,117,118],{},"20 GiB"," EBS volume was full — almost entirely from ClickHouse's internal query-profiling log with no TTL, while actual signal tables held only tens of megabytes.",[58,121,122,123,126,127,130,131,134],{},"I live-expanded the volume to ",[78,124,125],{},"50 GiB"," with no container restart, truncated bloated system tables, configured TTL retention on signal and system tables, and added a Terraform disk-usage alert that fires to Microsoft Teams at ",[78,128,129],{},"80%"," capacity. Combined with instrumentation across five services and ",[78,132,133],{},"22"," Terraform-managed alerts, that closed the blind spot where a healthy collector response masked a broken store.",[136,137],"case-study-cta",{"description":138,"primaryButtonText":139,"primaryButtonUrl":140,"secondaryButtonText":141,"secondaryButtonUrl":142,"title":143},"Looking to modernize your platform, build production-grade observability with OpenTelemetry and SigNoz, or instrument AI services with two-tier alerting? Let's talk.","START YOUR PROJECT","/contact","VIEW MY SKILLS","/resume","READY TO MODERNIZE\nWITH AI?",[145,146],"related-case-studies",{":limit":147,"current-slug":148},"3","trustclarity-observability",{"title":150,"searchDepth":151,"depth":151,"links":152},"",2,[153],{"id":50,"depth":151,"text":51,"children":154},[155,157,158],{"id":55,"depth":156,"text":56},3,{"id":63,"depth":156,"text":64},{"id":111,"depth":156,"text":112},"2026-03-26T00:00:00.000Z","md",false,{},true,"/case-studies/trustclarity-observability",0,{"title":167,"description":168,"ogTitle":169,"ogDescription":168,"keywords":170,"author":171,"articlePublishedTime":159,"articleTag":172},"OpenTelemetry SigNoz AWS ECS | Chicago","Chicago DevSecOps: OpenTelemetry and SigNoz on AWS ECS with Terraform alerts, ClickHouse hardening, and multi-service instrumentation.","OpenTelemetry and SigNoz Observability on AWS ECS","OpenTelemetry SigNoz AWS ECS, Terraform SigNoz alert automation, Distributed tracing OpenTelemetry ECS Fargate, Datadog to SigNoz migration, Infrastructure as code observability, DevSecOps & Cloud Observability, cloud observability consultant Chicago, Full Stack Engineer Chicago, TrustClarity, ClickHouse","Charles McGregory",[173,174,175,176,177,178,179],"OpenTelemetry","SigNoz","AWS ECS","Terraform","ClickHouse","DevSecOps","Chicago full stack","case-studies/trustclarity-observability",[173,174,175,176,177,182,183,184],"Node.js","Next.js","FastAPI","_3vRpO_NpA5f_j4sNTeCCZou-703fS6DiJNa2WT38Oc",[187,339,450],{"id":188,"title":189,"body":190,"date":316,"description":194,"extension":160,"featured":163,"image":199,"imageAlt":198,"impact":195,"meta":317,"navigation":163,"path":318,"rotation":165,"seo":319,"slug":308,"stem":331,"technologies":332,"__hash__":338},"caseStudies/case-studies/trustclarity-buyer-portal.md","TrustClarity: Buyer Portal Rebuild — Search-First TC 2.0 Platform",{"type":7,"value":191,"toc":309},[192,196,200,203,205,210,213,216,220,222,226,233,237,264,279,283,294,297,303,306],[10,193],{"description":194,"impact":195,"title":189},"React/Next.js rebuild of TrustClarity's core buyer workflows — Figma-to-production, WCAG 2.1 AA, Storybook/Chromatic visual gates, and a search-first architecture that replaced the per-org storefront layer.","400+ Storybook stories across 130+ components (from 11), WCAG 2.1 AA remediation, RFQ flow with server-side credential fix, and 1,533 lines of storefront code removed for a search-first pivot",[15,197],{"alt":198,"src":199},"Live TrustClarity buyer marketplace homepage with search-first navigation and promotional hero","/case-study-media/tc-marketplace-home.png",[20,201],{":metrics":202},"[{\"value\":\"400+\",\"label\":\"Stories\"},{\"value\":\"130+\",\"label\":\"Components\"},{\"value\":\"1,533\",\"label\":\"LOC cut\"}]",[24,204],{"project-label":26,"project-url":27},[29,206],{"role":207,"team":208,"timeline":209},"Full-Stack Frontend Engineer — buyer portal rebuild, design-system fidelity, accessibility, and visual regression gates","TrustClarity platform engineering","May–July 2026 (TC 2.0 buyer portal)",[35,211],{"challenge":212},"TrustClarity's TC 2.0 buyer portal needed a React/Next.js rebuild of core buyer workflows with Figma-level fidelity, WCAG 2.1 AA coverage, and hard visual-regression gates — while the product pivoted from per-org storefronts to a search-first experience. The surface also carried accessibility gaps, divergent add-to-cart rules across listing UIs, and an RFQ path that had exposed third-party credentials in the browser.",[39,214],{":results":215},"[\"Scaled Storybook/Chromatic visual testing from an 11-story baseline to 400+ stories across 130+ components\",\"Completed an 8-frame Figma design-token sync — Plus Jakarta Sans + Inter typography, category browse UI, and a footer rewrite from 3-column to 5-column\",\"Deleted the per-org storefront layer (1,533 lines) to lock in search-first architecture ahead of the component-library migration\",\"Shipped request-for-quote workflow; moved third-party writes server-side (closed credential exposure in the browser); unified buyability rules across listing surfaces; 70 unit tests (up from 46)\",\"WCAG 2.1 AA remediation — keyboard-nav comboboxes, ARIA form-error association, radiogroup semantics, skip links; patterns codified into shared accessibility standards; 575 lines of dead code removed\",\"Built buyer profile from Figma and fixed a silent auth-redirect bug so deep links survive login; 170/170 tests passing\"]",[43,217],{":technologies":218,"solution":219},"[\"React\",\"Next.js\",\"Vite\",\"Storybook\",\"Chromatic\",\"TypeScript\",\"shadcn/ui\",\"Figma\"]","Rebuilt TrustClarity's buyer portal as a search-first React/Next.js surface with Figma-to-production delivery, Storybook/Chromatic as hard visual gates, and WCAG 2.1 AA patterns codified into shared engineering standards. Removed the multi-tenancy storefront layer before the component-library migration, unified buyability across listing surfaces, and closed the RFQ credential exposure by moving third-party writes server-side.",[48,221,51],{"id":50},[53,223,225],{"id":224},"what-blocked-a-search-first-buyer-portal","What blocked a search-first buyer portal?",[58,227,228,229,232],{},"TrustClarity needed TC 2.0 buyer workflows — search, listing CTAs, request-for-quote, and auth-protected account surfaces — delivered as a modern React/Next.js app with production-grade design fidelity and accessibility. The prior per-org storefront model conflicted with a search-first product direction. Visual coverage was thin (an ",[78,230,231],{},"11-story"," Storybook baseline), accessibility semantics were incomplete, add-to-cart rules diverged across listing UIs, and the RFQ path had exposed third-party credentials in the browser.",[53,234,236],{"id":235},"how-was-the-portal-rebuilt-with-figma-fidelity-and-hard-quality-gates","How was the portal rebuilt with Figma fidelity and hard quality gates?",[58,238,239,240,243,244,247,248,251,252,255,256,259,260,263],{},"Visual testing scaled to ",[78,241,242],{},"400+ stories"," across ",[78,245,246],{},"130+ components",", with Chromatic as a hard PR gate so component drift showed up before it reached buyers. An ",[78,249,250],{},"8-frame"," Figma design-token sync drove typography (",[78,253,254],{},"Plus Jakarta Sans"," + ",[78,257,258],{},"Inter","), category browse UI, and a footer rewrite from ",[78,261,262],{},"3-column to 5-column",", with token normalization so Chromatic baselines matched the TC 2.0 design system.",[58,265,266,267,270,271,274,275,278],{},"The per-org storefront surface was no longer part of the product design. Deleting ",[78,268,269],{},"1,533 lines"," removed maintenance burden and Storybook noise before the component-library migration, locking the architecture to search-first routes. The RFQ workflow moved third-party writes to a server-side route so credentials never reached the browser. Listing and product-detail CTAs converged on shared buyability rules, with unit coverage growing from ",[78,272,273],{},"46"," to ",[78,276,277],{},"70"," tests. Accessibility remediation covered keyboard-navigable comboboxes, ARIA form-error association, radiogroup semantics, and skip links — patterns written into shared accessibility standards so later work reused the same rules.",[53,280,282],{"id":281},"what-shipped-for-buyers-and-engineering-quality","What shipped for buyers and engineering quality?",[58,284,285,286,289,290,293],{},"The buyer profile page was built from Figma (welcome banner, order history, account info, addresses). Mapping protected access surfaced a silent auth-redirect bug — unauthenticated deep links lost their return path and dumped users on the homepage after login. Consolidating redirect handling fixed every protected route; suite status at that milestone was ",[78,287,288],{},"170/170"," tests passing. WCAG 2.1 AA patterns and Chromatic gates remain the delivery bar, and ",[78,291,292],{},"575"," lines of dead story-only code were removed in the same accessibility pass.",[58,295,296],{},"The live marketplace reflects that search-first direction — category navigation, promotional merchandising, and a buyer search entry point on the public homepage:",[58,298,299],{},[300,301],"img",{"alt":302,"src":199},"Live TrustClarity marketplace homepage after the search-first buyer portal rebuild",[136,304],{"description":305,"primaryButtonText":139,"primaryButtonUrl":140,"secondaryButtonText":141,"secondaryButtonUrl":142,"title":143},"Looking to modernize a buyer-facing platform, tighten Figma-to-production delivery, or raise the bar on accessibility and visual regression gates? Let's talk.",[145,307],{":limit":147,"current-slug":308},"trustclarity-buyer-portal",{"title":150,"searchDepth":151,"depth":151,"links":310},[311],{"id":50,"depth":151,"text":51,"children":312},[313,314,315],{"id":224,"depth":156,"text":225},{"id":235,"depth":156,"text":236},{"id":281,"depth":156,"text":282},"2026-07-06T00:00:00.000Z",{},"/case-studies/trustclarity-buyer-portal",{"title":320,"description":321,"ogTitle":322,"ogDescription":321,"keywords":323,"author":171,"articlePublishedTime":316,"articleTag":324},"B2B Buyer Portal Next.js Case Study | Chicago","Chicago B2B SaaS engineer rebuilt a search-first buyer portal — Next.js, WCAG 2.1 AA React UI, and a Storybook design system with 400+ stories.","B2B Marketplace Buyer Portal Development — Next.js Case Study","B2B marketplace buyer portal development, Next.js search-first commerce architecture, WCAG 2.1 AA accessible React portal, Storybook design system Next.js, High-performance B2B catalog search, B2B SaaS Platform Architect, Full Stack Engineer Chicago, B2B SaaS engineer Chicago, TrustClarity, React, Chromatic, Figma",[325,326,183,327,328,329,330,179],"B2B marketplace","Buyer portal","React","WCAG 2.1 AA","Storybook","Search-first commerce","case-studies/trustclarity-buyer-portal",[327,183,333,329,334,335,336,337],"Vite","Chromatic","TypeScript","shadcn/ui","Figma","--6Wno34q0akfRYW9lta9pgvVFRoMKC2n08fHUTltJY",{"id":340,"title":341,"body":342,"date":429,"description":346,"extension":160,"featured":161,"image":351,"imageAlt":350,"impact":347,"meta":430,"navigation":163,"path":431,"rotation":165,"seo":432,"slug":421,"stem":442,"technologies":443,"__hash__":449},"caseStudies/case-studies/trustclarity-ai-multi-agent-delivery.md","TrustClarity: Orchestrating Google Jules Multi-Agent Delivery on the Buyer Platform",{"type":7,"value":343,"toc":422},[344,348,352,355,357,362,365,368,372,374,378,381,385,393,396,402,406,416,419],[10,345],{"description":346,"impact":347,"title":341},"Production orchestration of Google Jules on the TrustClarity buyer platform — first and second UAT rounds, signed-commit re-PR playbook, prompt discipline, and AGENTS.md governance that kept parallel cloud sessions safe to merge.","Four signed merges into develop covering AGENTS.md plus three UAT fixes; one additional thumbnail/search fix with a TypeScript no-suppression governance lesson; durable signed-commit / pull-and-re-PR playbook confirmed across later Jules rounds",[15,349],{"alt":350,"src":351},"Manga-style illustration of robot coding agents collaborating around floating pull-request panels","/case-study-media/tc-multi-agent.jpg",[20,353],{":metrics":354},"[{\"value\":\"4 PRs\",\"label\":\"Merged\"},{\"value\":\"Jules\",\"label\":\"Orchestration\"},{\"value\":\"Signed\",\"label\":\"Commits\"}]",[24,356],{"project-label":26,"project-url":27},[29,358],{"role":359,"team":360,"timeline":361},"Solo Full-Stack Consultant — Jules orchestration, prompt authoring, human review, signed re-commit, and repo governance","Solo engagement with Google Jules cloud sessions","First production run 2026-06-29; second UAT round 2026-06-30; learnings through 2026-07-08",[35,363],{"challenge":364},"TrustClarity needed a repeatable way to clear buyer-platform UAT failures without slowing the human roadmap. Google Jules can clone a GitHub repo in a cloud VM, implement a bounded fix, and attempt a PR — but sessions have no live secrets or runtime verification, cannot read Figma, and produce unsigned commits that signed-commit workflows cannot merge. Without file-scoped prompts, AGENTS.md conventions, and a human pull-and-re-PR runbook, parallel multi-agent batches risk scope creep, conflicting shared-file edits, and unreviewable merges.",[39,366],{":results":367},"[\"First production run (2026-06-29): four bounded UAT failures delegated to Jules from a 14-case buyer-platform UAT round\",\"Signed, hooks-passing merges into develop — AGENTS.md plus registration, category-slug, and seller-privacy fixes\",\"Required AGENTS.md sections committed before dispatch — stack, cloud-VM install notes, pre-PR check order, off-limits paths, ticketed PR titles, and pointers to shared standards\",\"Signed-commit runbook established: never merge the agent's PR; pull the diff → human review → ticket branch → signed commit with agent co-author credit → close unsigned PR (~5 minutes per fix)\",\"TypeScript no-suppression rule from a rejected cast on the thumbnail fix — structural typing only; reflected in shared standards and AGENTS.md\",\"Second UAT round (2026-06-30): ~17 issues swept; four slices dispatched to Jules; design-driven prompts required Claude to translate Figma into plain language\",\"Later batch confirmation: 13/13 sessions completed with 0 Jules-authored PRs — diffs real and on-target; pull-and-re-PR remains the durable operating model\"]",[43,369],{":technologies":370,"solution":371},"[\"Google Jules\",\"Claude\",\"GitHub\",\"AGENTS.md\",\"TypeScript\",\"React / Next.js\",\"Algolia\",\"pnpm\",\"Jira\"]","Orchestrated Google Jules as a cloud coding agent against the TrustClarity buyer platform while Claude authored file-scoped prompts with explicit root causes, in-scope and out-of-scope paths, and no-new-dependencies constraints. Each Jules session ran in a short-lived VM — clone, read AGENTS.md, implement, attempt PR — then a human pulled the diff, rejected type suppressions and scope creep, re-committed with a signed commit, opened a signed PR into develop, and closed Jules' unsigned PR. The first run dispatched unit/integration tests as a separate second wave after the fix PRs; the going-forward standard folds tests into the fix session. Prompt discipline also used parallel attempts for ambiguous bugs and single-line task prompts after multi-paragraph dispatch corruption.",[48,373,51],{"id":50},[53,375,377],{"id":376},"why-couldnt-cloud-coding-agents-merge-straight-into-the-buyer-platform","Why couldn't cloud coding agents merge straight into the buyer platform?",[58,379,380],{},"A UAT round on the TrustClarity buyer platform produced 14 test cases. Four failures were bounded, single-layer frontend fixes with clear root causes — registration payload missing a name field, wrong image key in search thumbnails, category slug mismatch between nav and search filters, and seller names leaking on buyer-facing surfaces. Google Jules can implement those kinds of fixes in a cloud VM, but sessions have no live secrets or runtime verification, cannot read Figma, and produce unsigned commits that signed-commit policies reject. Without file-scoped prompts and a human pull-and-re-sign runbook, parallel batches risked scope creep and unreviewable merges.",[53,382,384],{"id":383},"how-were-jules-sessions-orchestrated-safely","How were Jules sessions orchestrated safely?",[58,386,387,388,392],{},"Before any dispatch, ",[389,390,391],"code",{},"AGENTS.md"," was committed as the contract every cloud session reads first: tech stack, install notes that work in agent VMs (local git hooks break there), ordered pre-PR checks, naming conventions, off-limits paths, ticketed PR title format, and pointers to shared engineering standards.",[58,394,395],{},"Prompts named exact in-scope files, files not to touch, and the diagnosed root cause — not just the symptom. When two fixes both needed the shared search module, both sessions were told to read it but not edit it. Ambiguous bugs used parallel attempts; identical diffs across attempts counted as high confidence. Jules cannot read Figma — for design-driven work, Claude translated frames into plain-language layout and reuse instructions before dispatch.",[58,397,398,399,401],{},"Jules commits are unsigned. The durable runbook is: never merge the agent's PR; pull the diff; verify scope and reject type suppressions, unsafe casts, and lockfile edits; apply on a ticket branch; run checks; create a signed commit with agent co-author credit; open a signed PR; close the unsigned agent PR. That adds about five minutes per fix and keeps the signed chain unbroken. The TypeScript no-suppression rule came from rejecting a cast on the thumbnail fix in favor of structural typing — later reflected in shared standards and ",[389,400,391],{},".",[53,403,405],{"id":404},"what-did-production-runs-prove","What did production runs prove?",[58,407,408,409,411,412,415],{},"The first production run (2026-06-29) produced four signed, hooks-passing merges into develop — ",[389,410,391],{}," plus registration, category-slug, and seller-privacy fixes. The second UAT round (~17 issues) dispatched four slices to Jules and confirmed design-driven prompts need human-translated Figma context. A later batch confirmed the operating model at scale: ",[78,413,414],{},"13/13 sessions completed with 0 Jules-authored PRs"," — diffs were real and on-target; only PR-surfacing failed. Pull-and-re-PR is the durable path. Going forward, tests fold into the fix session instead of a separate second wave, and every tasks-file prompt stays on one physical line after multi-paragraph prompts silently corrupted into empty sessions.",[136,417],{"description":418,"primaryButtonText":139,"primaryButtonUrl":140,"secondaryButtonText":141,"secondaryButtonUrl":142,"title":143},"Looking to modernize delivery with multi-agent workflows, tighten AGENTS.md governance, or make AI-authored diffs safe to merge under signed-commit policies? Let's talk.",[145,420],{":limit":147,"current-slug":421},"trustclarity-ai-multi-agent-delivery",{"title":150,"searchDepth":151,"depth":151,"links":423},[424],{"id":50,"depth":151,"text":51,"children":425},[426,427,428],{"id":376,"depth":156,"text":377},{"id":383,"depth":156,"text":384},{"id":404,"depth":156,"text":405},"2026-06-29T00:00:00.000Z",{},"/case-studies/trustclarity-ai-multi-agent-delivery",{"title":433,"description":434,"ogTitle":435,"ogDescription":434,"keywords":436,"author":171,"articlePublishedTime":429,"articleTag":437},"Multi-Agent AI Coding Workflows | Chicago","Chicago AI engineering: Google Jules multi-agent delivery, signed-commit PR automation, and AGENTS.md guardrails for coding agents.","Multi-Agent AI Coding Workflows with Google Jules","Claude Code agent orchestration, Google Jules multi-agent delivery, AGENTS.md governance framework, Autonomous coding agent workflows, Multi-agent SDLC engineering, AI Engineering & Agent Governance, Full Stack Engineer Chicago, AI agent engineer Chicago, TrustClarity, Claude, GitHub",[438,439,391,440,441,335,179],"Multi-agent AI","Google Jules","Signed commits","Claude","case-studies/trustclarity-ai-multi-agent-delivery",[439,441,444,391,335,445,446,447,448],"GitHub","React / Next.js","Algolia","pnpm","Jira","QwBOd8PHA5UAFn7S9wTg3NQCPHigH-56Xi5oJB5syqM",{"id":451,"title":452,"body":453,"date":570,"description":457,"extension":160,"featured":161,"image":462,"imageAlt":461,"impact":458,"meta":571,"navigation":163,"path":572,"rotation":165,"seo":573,"slug":562,"stem":584,"technologies":585,"__hash__":586},"caseStudies/case-studies/trustclarity-supply-chain-security.md","TrustClarity: Supply-Chain Security Analysis and CI/CD Hardening",{"type":7,"value":454,"toc":563},[455,459,463,466,468,473,476,479,483,485,489,508,519,523,526,533,537,540,554,557,560],[10,456],{"description":457,"impact":458,"title":452},"Org-wide supply-chain security analysis for TrustClarity — malicious code found in 14 of 20 repositories across three attack vectors, remediated in repos before it reached TrustClarity's own production environment, followed by a SHA-pinned reusable CI/CD security pipeline and later hardening controls.","14 of 20 repositories remediated before TrustClarity production; 3 attack vectors eradicated; SHA-pinned reusable security-scan workflow rolled out to 20 repos; later SHA-pin and merge-base invisibility controls shipped",[15,460],{"alt":461,"src":462},"Manga-style illustration of sealed packages scanned by a heroic laser security gate","/case-study-media/tc-supply-chain.jpg",[20,464],{":metrics":465},"[{\"value\":\"14/20\",\"label\":\"Repos\"},{\"value\":\"3\",\"label\":\"Vectors\"},{\"value\":\"20-repo\",\"label\":\"CI\"}]",[24,467],{"project-label":26,"project-url":27},[29,469],{"role":470,"team":471,"timeline":472},"Solo Full-Stack Consultant — incident response, forensic reconstruction, remediation automation, and centralized CI/CD security pipeline design","Solo engagement with TrustClarity engineering stakeholders","Apr–May 2026 incident response and pipeline rollout; Jun–Jul 2026 hardening (nightly PR sweep, action SHA-pin scanner)",[35,474],{"challenge":475},"On 2026-04-29, TrustClarity discovered malicious code injected across its GitHub organisation. Fourteen of twenty repositories were infected over an approximately twelve-day window through three vectors: build-config injection, a JavaScript payload disguised as a Font Awesome font file, and a VS Code auto-exec launcher. The organisation needed rapid containment, history-safe remediation, and a centralized CI/CD security pipeline so the same gap could not reopen across every repo.",[39,477],{":results":478},"[\"Identified malicious code in 14 of 20 TrustClarity repositories across 3 attack vectors during a ~12-day injection window; forensic SHA reconstruction established the timeline and a repeatable incident-response standard\",\"Remediated infected repositories — malicious code removed from working trees and git history — before it reached TrustClarity's own production environment; leadership confirmed no customer data or production credentials were confirmed accessed\",\"Shipped an automated leftover scanner and restored branch protection after eradication\",\"Built a versioned, SHA-pinned reusable GitHub Actions security-scan workflow in the shared governance repo and rolled it out to 20 repos\",\"Closed a CVE-2025-30066-class mutable Actions-ref gap with a full-workflow pin scanner and shipped merge-base-invisibility detection plus a nightly open-PR sweep\"]",[43,480],{":technologies":481,"solution":482},"[\"GitHub Actions\",\"Semgrep\",\"Gitleaks\",\"SonarCloud\",\"Node.js\"]","Led TrustClarity's org-wide supply-chain response — mapping three injection vectors, remediating infected repositories before TrustClarity's own production environment was implicated, and turning the lessons into prevention. A reusable, SHA-pinned GitHub Actions workflow in the shared governance repo became the org standard, installed across 20 repos. Later hardening added full-workflow action SHA-pin scanning and nightly head-tree scans of open PRs so merge-base-invisible payloads cannot hide in GitHub's default diff view.",[48,484,51],{"id":50},[53,486,488],{"id":487},"what-was-the-supply-chain-injection-problem","What was the supply-chain injection problem?",[58,490,491,492,495,496,499,500,503,504,507],{},"In late April 2026, TrustClarity's GitHub organisation was hit by a coordinated supply-chain injection. Across a roughly twelve-day window, malicious code landed in ",[78,493,494],{},"14 of 20"," repositories through ",[78,497,498],{},"three vectors",": obfuscated injection inside a build-tool config file, a JavaScript payload disguised as a Font Awesome font file (",[389,501,502],{},"fa-solid-400.woff2","), and a VS Code task launcher (",[389,505,506],{},".vscode/tasks.json",") that auto-ran the payload when a developer opened the project. Forensic SHA reconstruction established the timeline, and the response steps were written down as a repeatable incident standard.",[58,509,510,511,514,515,518],{},"The catch is intentionally narrow. Malicious code was discovered and remediated in repositories ",[78,512,513],{},"before"," it reached TrustClarity's own production environment. Per the leadership summary, ",[78,516,517],{},"no customer data or production credentials were confirmed accessed",". This case study does not claim a production breach, customer impact, or runtime compromise.",[53,520,522],{"id":521},"how-was-the-incident-contained-and-turned-into-prevention","How was the incident contained and turned into prevention?",[58,524,525],{},"Remediation cleared all three vectors from working trees and rewrote git history where payload blobs or cover scaffolding remained in past commits. An automated leftover scanner confirmed a clean post-remediation state, and branch protection was restored so force-push windows used during cleanup could not stay open.",[58,527,528,529,532],{},"Detection alone was not enough — the same inconsistency that let the incident span many repos would return if every team configured scanners differently. I shipped a versioned, SHA-pinned reusable GitHub Actions workflow in the shared governance repo that runs the org's security-scan pipeline — including Semgrep, Gitleaks, and SonarCloud-facing checks — from one maintained source. Day-0 distribution opened adoption PRs across ",[78,530,531],{},"20 repositories",", with consumer stubs pinned to immutable tags rather than mutable branch refs.",[53,534,536],{"id":535},"what-later-hardening-closed-remaining-gaps","What later hardening closed remaining gaps?",[58,538,539],{},"Two control gaps surfaced after the initial rollout and were closed with follow-on tooling:",[72,541,542,548],{},[75,543,544,547],{},[78,545,546],{},"Mutable Actions refs (CVE-2025-30066-class):"," An unpinned third-party Action slipped past checks that only validated the reusable-workflow caller pin. A pin scanner now walks every workflow file and fails on third-party actions that are not full commit SHAs.",[75,549,550,553],{},[78,551,552],{},"Merge-base invisibility:"," A PR branch created before a remediation commit can hide payload lines from GitHub's default diff. A nightly open-PR sweep materializes each PR head without running hooks and runs the security pipeline against that tree.",[58,555,556],{},"Together with on-push security-scan coverage, TrustClarity gained complementary layers that watch protected branches, open PR heads, and Action pin hygiene — not only the single stub line that first adoption checked.",[136,558],{"description":559,"primaryButtonText":139,"primaryButtonUrl":140,"secondaryButtonText":141,"secondaryButtonUrl":142,"title":143},"Looking to harden supply-chain controls, centralize CI/CD security scanning, or modernize how your team catches repo-side threats before production? Let's talk.",[145,561],{":limit":147,"current-slug":562},"trustclarity-supply-chain-security",{"title":150,"searchDepth":151,"depth":151,"links":564},[565],{"id":50,"depth":151,"text":51,"children":566},[567,568,569],{"id":487,"depth":156,"text":488},{"id":521,"depth":156,"text":522},{"id":535,"depth":156,"text":536},"2026-04-29T00:00:00.000Z",{},"/case-studies/trustclarity-supply-chain-security",{"title":574,"description":575,"ogTitle":576,"ogDescription":575,"keywords":577,"author":171,"articlePublishedTime":570,"articleTag":578},"Software Supply-Chain Security CI/CD | Chicago","Chicago DevSecOps: supply-chain hardening with SHA-pinned GitHub Actions, Semgrep, Gitleaks, and SonarCloud across twenty repos.","Software Supply-Chain Security and SHA-Pinned CI/CD","SHA-pinned GitHub Actions workflows, CI/CD supply-chain security hardening, Semgrep automated rule enforcement, Gitleaks secret scanning CI/CD, Reusable GitHub Actions security patterns, DevSecOps & Cloud Observability, DevSecOps consultant Chicago IL, Full Stack Engineer Chicago, TrustClarity, SonarCloud",[579,580,581,582,583,178,179],"Supply-chain security","GitHub Actions","Semgrep","Gitleaks","SonarCloud","case-studies/trustclarity-supply-chain-security",[580,581,582,583,182],"7HIKNyB4eSS4WnVovSMmGcCMhqstKLp-x35T-jVvbPw",1791761132106]