Recovering Gracefully From a Failed Daily Publish

A daily publishing run fails after it sends a merge request. The automation sees a timeout, but it does not know whether GitHub accepted the merge. The next scheduled run is approaching. Retrying every step feels decisive; it is also how one intended article becomes two pull requests, two deploys, or two live URLs.

Graceful recovery begins by treating failure as an uncertain system state, not as permission to start over. The goal is not to make every stage green retroactively. It is to establish what happened, preserve the editorial decision, repair the smallest failed boundary, and converge on exactly one correct reader-visible outcome.

That distinction matters in an agent-operated studio. Speed comes from allowing routine work to proceed autonomously. Trust comes from stopping when the system cannot prove that its next action is safe.

Define the intended outcome before diagnosing the attempt

A failed run is easier to recover when its intended outcome was recorded before execution:

• one scheduled topic and date;
• one canonical slug and content path;
• one approved article revision;
• one content branch and pull request;
• one merge into the protected branch; and
• one verified production URL exposed through the blog index, sitemap, and RSS.

This recovery target is more useful than a generic status such as pipeline failed. A pipeline can report failure after the intended merge succeeded, or report success while production still serves stale content. Recovery must compare observed state with the intended outcome at every durable boundary.

Begin with the run identifier and expected artifact. Do not generate a new slug, draft, or run identity merely because an execution attempt ended. A new attempt should remain attached to the same publishing intent unless an editor explicitly replaces the article.

Contain the failure before changing anything

The first response should reduce the chance of compounding the incident.

Pause automatic retries that can create external effects. Coalesce the next scheduler tick into the active recovery instead of launching overlapping work. Preserve the branch, commit, pull request, check results, deployment references, and production observations. Mark uncertain actions as not confirmed rather than guessing that they failed.

Then state the reader-visible impact precisely. Useful descriptions include:

• the article is not merged and is not live;
• the merge is confirmed, but production still serves the previous revision;
• the article route is live, but the blog index or feed does not include it;
• the article is live twice under different URLs; or
• the outcome cannot yet be determined because the provider response was lost.

“Publishing is broken” is too broad to guide a safe response. Containment should identify which effects must stop and which evidence must be preserved.

Google's Site Reliability Engineering guidance recommends beginning troubleshooting with a problem report, examining the system end to end, and changing one variable at a time [1]. Those principles fit publishing incidents well: define the observed failure, trace the complete path, and avoid simultaneous edits that obscure the cause.

Reconcile durable state before retrying

An error response describes one observation, not necessarily the final state of the system. Before retrying, query each system that can establish a durable fact.

Editorial state

Confirm the exact article revision that passed review, its QA score, its sources, and the approval threshold. If the draft changed after QA, the previous score no longer authorizes publication. Recovery must return the changed artifact through validation and editorial review.

Repository state

Resolve the content path, branch head, commit SHA, pull request state, merge commit, and required checks. Search the protected branch for the expected slug and workflow run identifier. If the content is already present, do not open another pull request.

GitHub supports rerunning all jobs, failed jobs, or a specific job while retaining the original commit SHA and Git ref for the rerun [3]. That makes a failed required check recoverable without inventing a second code change. Use a rerun only after deciding that the failure was transient and the unchanged commit remains valid.

Deployment state

Determine which commit production currently serves. A failed optional preview is different from a rejected protected-branch merge, and a successful deployment event is different from verified page content. Preserve provider identifiers, but make the repository's required checks and merge result the publication gate.

Production state

Inspect the canonical article route and its discovery surfaces from the outside. Verify the title, publication date, canonical link, description, and a stable body excerpt. Confirm that /blog/, the sitemap, and RSS each expose exactly one matching URL. A 200 response alone is not enough; it can return stale or incorrect content.

Write the reconciliation result as a small state table or structured record. For every boundary, distinguish confirmed absent, confirmed present, failed, and unknown. Unknown is a safe state because it prevents an unverified assumption from becoming another mutation.

Classify the failure at the boundary that owns it

Once durable state is known, classify the failure by where recovery must occur.

Research or editorial failure: a source is unverifiable, required metadata is missing, the article duplicates existing coverage, or QA is below threshold. Return to the relevant content stage. Do not use deployment retries to bypass a content stop.

Validation failure: the canonical file exists but violates the schema, route rules, or build contract. Correct the same content branch, rerun the focused checks, and obtain a new QA decision if the correction changes meaning.

Repository failure: the commit is valid but a required check failed, the branch is behind, conflicts exist, or GitHub rejects the merge. Repair within the existing pull request when practical. Protected branch rules can require status checks, reviews, signed commits, linear history, or successful deployments before merging [4]. Recovery respects those controls; it does not weaken or bypass them.

Deployment failure: the merge is confirmed but the production deployment failed or stalled. Retry or repair the deployment for the confirmed merge commit. Do not create another article commit merely to trigger infrastructure.

Verification failure: deployment completed, but the article or one of its discovery surfaces is wrong. Treat this as a production defect. Decide whether the smallest safe response is a corrective commit, rollback, or feed/index repair, and verify the complete bundle again.

Ambiguous external action: the request timed out and its effect is unknown. Reconcile against the provider before issuing the action again. This is the boundary where disciplined recovery prevents the most duplicate work.

Classification keeps ownership clear. It also stops a visible symptom—such as a missing RSS item—from causing unrelated research or drafting to repeat.

Resume from the last trustworthy checkpoint

Recovery should reuse every artifact whose identity and validity remain provable.

If research and QA passed for the exact commit, retain them. If validation passed but a required CI job failed from a transient runner outage, rerun that job against the same commit. If GitHub accepted the merge but the response was lost, record the confirmed merge and continue with deployment verification. If production is correct but the final verification record failed to save, reconstruct the record from observable evidence rather than republishing.

A safe recovery sequence is:

1. freeze duplicate-producing retries;
2. capture the intended topic, slug, revision, and run identity;
3. reconcile editorial, repository, deployment, and production state;
4. identify the earliest untrustworthy checkpoint;
5. fix the smallest owning boundary;
6. rerun every downstream check invalidated by that change;
7. proceed through the ordinary protected merge path when a merge is still needed; and
8. verify one live canonical outcome across all discovery surfaces.

Changing content invalidates content validation and may invalidate QA. Changing only the retry mechanism does not justify rewriting the article. The recovery record should explain which evidence was reused and why it remained valid.

Keep branch protection inside the recovery path

Incidents create pressure to treat controls as obstacles. That is exactly when those controls are most valuable.

Never use an administrative or force merge to recover a daily article. Never mark a required check optional during the incident. Never rewrite protected history to make the record look clean. If GitHub reports that the pull request is not mergeable or rejects the ordinary merge, publication remains blocked until the stated condition is resolved.

GitHub documents that protected branches can require passing status checks and other conditions before merge [4]. Those requirements provide a shared, inspectable decision boundary. An optional hosting preview may fail while GitHub still considers the pull request mergeable; the repository's actual rule set decides whether the normal merge can proceed. Recovery should record that distinction instead of treating every external status as either universally blocking or safe to ignore.

Close with production proof, not a green workflow

The recovery is complete only when the intended article is correct in production and the record explains how that conclusion was reached.

Verify:

• the canonical article URL returns the expected content;
• metadata matches the approved package;
• /blog/ lists the article once with the correct date;
• the sitemap contains one canonical URL;
• RSS contains one matching item; and
• the live artifact corresponds to the confirmed protected-branch commit.

Record the original failure, the classification, the corrective action, reused evidence, rerun results, merge commit, pull request, production URL, and verification time. If any assertion remains unknown, the run remains blocked. A recovery note should never convert missing evidence into a success label.

Learn without rewriting history

After service is restored, capture what would make the next incident shorter or safer. Google SRE describes blameless postmortems as a way to document impact, trigger, response, root causes, and follow-up actions so an organization can learn from failure [2]. The useful question is not which agent or person caused the stop; it is which system condition allowed ambiguity or delayed diagnosis.

Small improvements often have the greatest effect:

• persist the intended slug and commit before external mutations;
• distinguish failed from not confirmed;
• attach provider identifiers to the run record;
• make retries idempotent and bounded;
• test the article route and discovery surfaces as one verification bundle;
• name the owner and next action on every blocked outcome; and
• add an automated assertion for the exact condition that escaped.

Do not hide the failed attempt or squash away evidence merely to produce a tidy narrative. The history is operational data. It shows which checkpoint failed, which action restored progress, and whether safeguards behaved as designed.

Recovery is controlled convergence

A resilient publishing system does not promise that every daily run will succeed on its first attempt. It promises that failure will not create uncontrolled side effects or pressure the operator to guess.

Contain the run. Reconcile durable state. Classify the owning boundary. Resume from the last trustworthy checkpoint. Keep protected-branch rules intact. Finish with externally verified proof that one approved article reached readers through one canonical path.

That is graceful recovery: not erasing the failure, but using evidence to make the next action safe.

Sources

1. Google, Site Reliability Engineering, “Effective Troubleshooting”: https://sre.google/sre-book/effective-troubleshooting/
2. Google, Site Reliability Engineering, “Postmortem Culture: Learning from Failure”: https://sre.google/sre-book/postmortem-culture/
3. GitHub Docs, “Re-running workflows and jobs”: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs
4. GitHub Docs, “About protected branches”: https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches