What a week of AI-assisted website changes taught us

The useful changes this week came from finding where our assumptions were wrong. A working release did not guarantee a useful answer. A source link did not guarantee relevant evidence. Missing traffic data did not mean nobody visited.

This review covers 18–24 October 2026, with the 16–17 October baseline for context. The trial is still running. Its saved hard stop is 25 October at 1:45 pm fixed GMT+10 — 2:45 pm Sydney time. This is an interim review, not a claim that the final activations have happened. No schedule is extended here.

Three different kinds of evidence

• Delivery: saved releases show GitHub checks, Cloudflare publication and live verification. Individual unattended runs are recorded; the whole week was not intervention-free.
• Answer relevance: controlled questions exposed failures that needed repairs. Passing those examples does not establish general accuracy.
• Reader benefit: no completed reader study is recorded. Traffic, votes and releases do not establish understanding.

The earlier learning article explains the delivery-versus-benefit gap. This review asks what it changed in how we operate.

Seven lessons to carry forward

1. Remove machinery before adding more

The 18 October record documents the GitHub-to-Cloudflare release path and retirement of the Linux runner, publisher and monitoring. Useful history was preserved privately.

One roadmap, one active policy and one daily journal reduced scattered instructions. Analyst, planner, operator and auditor are responsibilities within a run, not four independent reviewers. Keep the smallest system that can make, publish and explain a decision.

2. Some questions do not need AI

“What’s new?” initially favoured keyword-heavy older records. Even with the newest source available, the model could ignore it. The 21 October record documents the simpler answer: quote the published update with its date and link, without inference.

Use dates and records for deterministic questions. Save model calls for work that needs them.

3. A citation can be relevant to the website and wrong for the question

A privacy question returned keyword-search storage text. A how-to answer lost its steps. A reader-study question could select a builder recipe.

The 22 October and 23 October records show focused repairs: keep procedures together, prefer current evidence and display validated published passages. The model selects wording rather than supplying an unchecked displayed paraphrase.

Check the actual passage against the question. Keep varied examples and source fallbacks; do not turn a few successful examples into an accuracy claim.

4. Missing measurement needs investigation

An owner challenge led us beyond overview cards to loaded detailed reports, matching windows and filters. Browser visits and Function invocations describe different things.

The 24 October snapshot reports 11 browser-recorded visits and 20 page views over seven days. Its detailed 24-hour reports say “No data”. Three Yes responses include at least two known tests. These are not verified people, independent endorsements or conversion results.

Keep the source, window, tests and uncertainty beside each figure. Do not convert missing data into zero or add unlike measures together.

5. Security has an operating cost

The 20 October record documents Turnstile, bounded requests, caching, off switches and a shared daily AI reservation limit. Static reading and keyword search remain available without optional AI.

Older retained endpoints and distributed quota exhaustion still need assessment. Existing tests do not prove complete protection. Total trial cost and owner time have not been established. Keep publishing credentials away from visitor answers, fail closed on verification or budget uncertainty, and understand the workload before buying capacity.

6. Accessibility needs interaction checks

The 23 October review found unclear short-question errors and a fixed-height Ask button that overflowed with enlarged text. The 24 October check found Roadmap overflow with wider spacing.

Those cases received targeted fixes. They do not establish full WCAG conformance. Screen-reader speech, complete flows and third-party verification remain outstanding. A passing build is not an accessibility assessment.

7. Activity is useful when it changes a decision

Homepage iterations produced clearer actions and activity rings. The journal became the rings’ single source, with unknown coverage preserved. Later audits removed repeated copy.

There is no reader evidence yet that the rings or shorter copy improved understanding. More activations can create more material to maintain. Permit justified no-change, removal and rollback decisions. Record what evidence changed the approach; publishing every time is a poor success measure.

The next operating model

1. Finish and reconcile the trial. Confirm outstanding releases, preserve failed and no-change outcomes, and honour the saved stop. Assess delivery, reader evidence, usage and manual interventions separately.
2. Propose one daily activation, not four. If authorised, try a further 14-day phase at 5 am fixed GMT+10. Put weekly review inside that run, rather than adding a reporting schedule.
3. Choose from evidence. Inspect live state, unresolved work and limits; assess eligible ideas; name the reader task and check. Implement the useful batch without a fixed item count or forced change.
4. Record the learning. Save observation, hypothesis, decision, result and next check in existing records. Change priorities when evidence supports it. The underlying model is not training itself.
5. Escalate exceptions. Stop the affected action for expired authority, uncertain publication, failing checks, credentials or limits. Routine authorised content need not wait for approval; spending, new collection and wider access still do.
6. Review outcomes weekly. Count verified releases, failures, blocked attempts and owner interventions. Assess recurring reader-task problems and actual usage where available. Continue, change direction or stop from evidence, not output volume.

This is a recommendation, not an activated schedule. The current Windows workflow still needs the computer, app, network and credentials available. Browser reporting is not an unattended analytics API. A hosted replacement needs its own access, cost and recovery rehearsal; local sign-ins do not automatically transfer into a Codex cloud environment.

Before expanding autonomy

• Finish retained-endpoint protection and service-usage assessment.
• Rehearse recovery from a failed release. Rollback is not proven merely because a procedure exists.
• Use the five-minute reader check with willing people. Automation cannot invent participants or understanding.
• Verify unattended credentials and evidence access. Keep unavailable metrics unknown.
• Agree next-phase dates, publishing scope and limits before enabling it. A proposed domain purchase does not authorise migration.

Self-improvement here means a traceable change of decision: an observation revises an assumption and makes the next action better justified. The release loop can run automatically within its boundaries. Whether those decisions help readers remains a question to test.

Later owner decision · 24 October

The owner chose to retain four daily activations and extend the existing trial by 14 days, through 8 November. This supersedes the proposed daily cadence above. Weekly learning articles use those existing runs, including the question: what would we do differently starting again? See the current trial terms and dated decision.