Skip to content

Mark posts deleted on the network, and send webhooks when they go or come back - #464

Merged
paulocastellano merged 46 commits into
trypostit:mainfrom
mrbeandev:fix/detect-deleted-meta-posts
Nov 9, 2026
Merged

paulocastellano merged 46 commits into
trypostit:mainfrom
mrbeandev:fix/detect-deleted-meta-posts

Conversation

@mrbeandev

@mrbeandev mrbeandev commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Refs #228. Closes #454. Closes #484. Thanks to @mrbeandev for finding this and testing it on a real account; this PR builds on that work.

The problem

When a post TryPost published is deleted on the network, TryPost keeps showing it as published, with a "View post" link to a page that no longer exists, and the post cannot be deleted in TryPost (published posts are protected).

The change

Detection, on every network that reads a post again after publishing. The analytics metrics read is the only time TryPost reads a published post again, so detection rides on it: no new job, and an extra request only once a read already failed. A post is marked deleted only when the collector confirms it (AnalyticsCollectionException::deletedOnNetwork()): the network's documented "does not exist" answer for the post, and, where that answer could also mean lost access, an account-level read with the same token (the list its discovery already reads) still works, so an expired token or a lost permission is never taken for a deletion. A confirming read that fails only for now (rate limit, server error) is rethrown, so the job retries instead of settling without a confirmation.

Network Confirmed by
Facebook, Instagram, Threads Graph code 100 / subcode 33 for the post, while /{account}/published_posts, /media or /threads still reads. A Facebook video or reel read without insights is checked by reading the video, but only once its reads were refused, never on a Page that did not grant pages_read_engagement. Stories are never checked: they expire on their own.
YouTube videos.list (owner token) no longer returns the video, read by its own request, never from another video's cached batch.
X The documented resource-not-found problem naming the post (resource_type tweet, its resource_id), in a 200's errors or a 404's body, also on the thread lookup with reply polls. No extra request (X bills each one).
Bluesky getPosts leaves the post out, and its repository (the account's PDS, self-hosted included) answers RecordNotFound while listRecords still reads (the AppView also hides moderated posts).
Mastodon A 404 or 410 on an owner read (read scope; without it a 404 also means private), while the account's statuses still read.
Pinterest A 404 on the Pin analytics and on the Pin, while the Pins list still reads.

Any other 404 or 410 keeps marking only the analytics publication, as before.

Not covered: TikTok's API cannot tell a deleted video from a private one (and TikTok has no delete endpoint, so a wrong "deleted" could not be undone). LinkedIn, Google Business, Telegram and Discord have no read after publishing; #481 specs the networks that can be covered with a new periodic read and the ones that cannot.

Where it is stored. A new PublishStatus::Deleted (publish_status = deleted), the network side of the post; status stays published. No new column. CollectPublicationMetrics writes it with Post::markPublicationDeleted() under the publication's row lock, unless a sync saw the post since the second the read started (last_seen_at at or after it; a sync earlier in that same second only delays the deletion to the next read). A sync that lists the post again restores it (markPublicationRestored()), without rerunning the "just published" side effects, and queues a fresh metrics read. A discovery reads the publication's post only when the publication was marked deleted, so a normal sync costs no extra query.

A post deleted on the network (Post::isDeletedOnNetwork()):

  • can be deleted in TryPost on the web, the API and MCP;
  • finalizes as published if finalize runs late;
  • still counts as published for the posting goal, the lifecycle stage and PostHog activity (PublishStatus::wentOut(), Post::scopeWentOut()), while live-only reads (Instagram grid, analytics sync, the importer, TikTok ids) keep scopePublicationPublished();
  • shows publish_status: deleted in API, MCP and webhook payloads, and its metrics answer reason: deleted_on_network.

Webhooks. Two new events, post.deleted_on_network and post.restored (EventType::fromPublishStatus()), listed in the webhook settings, the MCP tool and all 16 locales. A post imported from the network sends neither, like it never sent post.created or post.published (ImportExternalPosts). Both transitions also broadcast post.platform.status.updated, so an open schedule reloads, imported posts included. The transition is written in a transaction, so the webhook never reads the post while writePublication() has its timestamps off.

UI. The card shows a Deleted badge; clicking it opens a popover ("Deleted on Instagram" / "It was removed after it was published, so its link and insights are no longer available."), like the failure popover. The network link is gone from the card, the menu (Share / Copy link) and the details dialog, whose header and channel rail show Deleted instead of Published. The calendar chip shows a Deleted mark (calendar.post_status.deleted) and the repurpose activity list labels it too. Insights keep the post's historical numbers and drop its network link in the channel table, the report's top posts and the export (PublicationAvailability::visiblePermalink()). Strings in all 16 locales.

Cleanups along the way.

Tests

  • Collectors, per network: deleted when confirmed; not deleted when the account list fails, when the post still reads, on a server error or a refusal, on a different error code, on a permission or authentication refusal (no extra request), for a story, for X's not-authorized-for-resource, another post's or a non-tweet problem, or one without resource_id, for a Bluesky record the AppView hides or a deactivated repository, for a Mastodon read without the owner token, for a Pinterest Pin that still reads or is refused, and for YouTube with a cache that keeps nothing or a stale batch. A Bluesky post that is present is read once, also on a self-hosted PDS; the exact account reads (and Mastodon's token) are asserted; a confirming read that fails only for now is retried on every network; Meta only checks after a malformed read and forgets refusals between posts; a Facebook reel without insights is checked by reading the video once, and never on a Page that did not grant reading it.
  • Job: a confirmed deletion moves the post for YouTube, X, Bluesky, Mastodon and Pinterest, a bare X 404 does not; a publication removed during the check is left alone; a sync during the check keeps it published, even in the same second; only a published post is marked and only a deleted one restored; restoring dispatches no "just published" job; a discovery of a live publication never reads its post.
  • Webhooks: deleted and restored send their event with the post payload and broadcast the refresh; restoring never sends post.published; an imported post sends no webhook but still refreshes; EventType::fromPublishStatus() maps every transition.
  • Insights: the channel table, the report's top posts and the export keep a deleted publication without its link.
  • Config: a URL set with a trailing slash comes back without it, an unset one keeps its default, and loading the config in a test puts the environment back; both channel-server accessors trim and fall back to the config.
  • Rules, finalize, posting goal, PostHog activity and lifecycle, web/API/MCP deletion and metrics reason parity, Sent tab props.
  • Browser: card badge and popover, menu (no share or copy link), details dialog header and rail (deleted and live siblings), calendar chip mark, a publishing post never shown as deleted, and the repurpose activity list (Deleted and Google review labels).
  • Every guard was removed once to check its test fails. Pint, ESLint and vue-tsc are clean; the full suite passes on PostgreSQL (11,033 tests) and the analytics and post deletion tests on MySQL.

Manual test

Plan: reset the database, create a new account, connect every network and check the analytics pull real data from scratch; then publish from TryPost, delete the post on the network and run the metrics job for that publication (CollectPublicationMetrics). Expect the Deleted badge and popover, no "View post", and Delete in the menu.

  • Analytics from scratch: every connected network imports its posts and fills their metrics and followers.
  • Instagram via Facebook
  • Instagram Login
  • Facebook Page post and Reel
  • Threads (not yet confirmed that Threads answers 100/33 for a deleted post; otherwise nothing changes)
  • YouTube
  • X
  • Bluesky
  • Mastodon (account connected with read)
  • Pinterest
  • No false positive: with a post that still exists, revoke a permission or disconnect the account, run the job: the post stays published.
  • Story: delete an Instagram or Facebook story before it expires and run the job: it stays published.
  • Details dialog: shows Deleted instead of Published, and no link to the network.
  • Calendar: the chip shows the Deleted mark.
  • Insights: the deleted post keeps its numbers in the channel table, without a link.
  • Webhooks: subscribe to both new events; deleting the post on the network sends post.deleted_on_network.
  • Duplicate from the deleted post's menu opens a draft with the same content.
  • Repurpose: a repurposed post deleted on the network shows "Deleted" in the activity list.
  • API / MCP: GET /posts/{id} shows publish_status: deleted, metrics show reason: deleted_on_network, and deleting it works.
  • Mobile (390px): the badge fits on the card.

After deploy

YouTube publications the metrics read already marked deleted before this PR keep their post as published. To see how many there are (read only):

SELECT ap.platform,
       COUNT(*) AS publications_deleted,
       COUNT(p.id) FILTER (WHERE p.status = 'published' AND p.publish_status = 'published') AS posts_still_published
FROM analytics_publications ap
LEFT JOIN posts p ON p.id = ap.post_id
WHERE ap.availability = 'deleted'
GROUP BY ap.platform;

Graph answers code 100, subcode 33 when a post was deleted on Instagram or
Facebook after publishing, but the Meta error mapping filed it as a generic
malformed read, so the publication stayed available and the post card kept
showing it as published with a link to a post that no longer exists.

The Instagram and Facebook metric collectors now read the post itself when a
metrics read fails as malformed. If Graph says the post does not exist while
the account's own node still reads with the same token, the post is gone and
the publication is marked deleted. The same subcode also means "missing
permission", so a token that cannot read the account is never treated as a
deletion. Threads is unchanged.

A published post whose publication is deleted gets a Deleted badge, a note,
Delete and Repost buttons on its card, and may be deleted in TryPost.

Refs trypostit#228
@paulocastellano

Copy link
Copy Markdown
Contributor

Thank you @mrbeandev i will check it!

Paulo Castellano added 2 commits November 4, 2026 09:58
…ta-posts

# Conflicts:
#	tests/Feature/Analytics/Collectors/PublicationMetricsCollectorsTest.php
#	tests/Feature/Analytics/PublicationMetricsJobsTest.php
…hreads

- Add PublishStatus::Deleted. When the metrics read finds a published post
  gone (YouTube, or Facebook, Instagram and Threads), the post moves to
  publish_status deleted (status stays published) and can be deleted in
  TryPost. A discovery that lists it again restores it.
- Move the Meta deletion check into AbstractMetaPublicationMetricsCollector
  for all three networks. Instagram Login and Threads read `me`; Instagram
  through Facebook reads the Instagram account, not the Page; Facebook reads
  the Page. A story older than a day is never reported as deleted.
- Stop loading the analytics publication on every card (it sent the whole
  row to the browser); the deletion rule reads publish_status only.
- Hide the network link on the card, the card menu and the details dialog.
  Drop the duplicate Delete and Repost buttons: the card menu already has
  Delete and Duplicate. Move the strings to posts.publish.deleted_on_network.
- Tests: config URLs in fakes, no fixed dates, Threads and Instagram Login
  cases, web/API/MCP deletion parity, sent tab props, a browser test of the
  card, and the badge's single-line budget.
@paulocastellano paulocastellano changed the title Mark Instagram and Facebook posts deleted on the network Mark posts deleted on Facebook, Instagram and Threads Nov 4, 2026
Paulo Castellano added 7 commits November 4, 2026 10:46
- Only a deletion the collector confirms moves the post: YouTube, or Meta
  through AnalyticsCollectionException::deletedOnNetwork(). A bare 404 or 410
  from other networks only marks the analytics publication, as before.
- Meta checks the account's posts edge (published_posts, media, threads), the
  one discovery reads, instead of the public account node. Stories are never
  checked, since they expire on their own.
- Restoring a post no longer reruns the "just published" side effects.
- A post still publishing can't be deleted; a deleted post finalizes as
  published; the posting goal, lifecycle and PostHog activity keep counting it
  (Post::scopeWentOut()).
- Front end: isDeletedOnNetwork() and postCardPermalink() in
  usePostCardActions guard the card, menu, details dialog and share/copy; the
  details dialog shows the Deleted badge; the repurpose list labels the state
  instead of showing a raw key; the badge is subtle and the hint skips Delete
  for members who can't delete.
- Translations follow each locale's menu words (fr, el, es, nl, uk).
- Tests for every guard, each checked to fail without it.
- Remove the outdated DEPLOYMENT.md (trypostit#454).
The note under the post moves into a popover on the Deleted badge, like the
failure popover. The details dialog shows Deleted in place of Published, so a
post shows one status.
Post::isDeletedOnNetwork() replaces PostStatusRules::deletedOnNetwork() and the
check in ReadPublicationAnalytics. The popover drops the account line: the card
already shows which channel it is.
The title says where the post was deleted; the text says what that changes
(no link, no insights) instead of repeating it.
- The details dialog's channel rail shows a deleted sibling as deleted.
- YouTube reads its statistics cache once, so an entry expiring between two
  reads can no longer look like a deleted video.
- Meta checks whether a post was deleted only after a read answered "object
  does not exist", saving a Graph request on live posts whose metrics are
  refused for another reason.
- The metrics job marks a publication and its post deleted under the row
  lock, and not when a sync saw the post while it was being checked.
- PublishStatus::wentOut() is the one definition of "published or deleted",
  used by the scope and by FinalizePostPublication.
- PostDeletedOnNetworkBadge is shared by the popover and the details dialog;
  RepurposeItemList uses transform(); Italian uses "insight".
- post.md and AGENTS.md describe the final rules.
@mrbeandev

Copy link
Copy Markdown
Contributor Author

I ran the manual test list against eafc1890 on a self-hosted instance with real accounts (Instagram via Facebook login and a Facebook Page, both the same Byte Sized account, a Meta app in development mode). For each case I published from TryPost through the API, deleted the post in the network's app, then ran CollectPublicationMetrics.

One thing to know first: the job only re-reads a publication that has no snapshot for that day (refreshSameDay is false), and a new post gets its first read an hour after publishing. So the first run on the same day did nothing; I reran with refreshSameDay: true.

Results

Case Graph answer for the deleted post After the job
Instagram via Facebook, reel 100 / 33 publish_status: deleted, card shows Deleted, deletable
Facebook Reel 100 / 33 publish_status: deleted, deletable
Facebook Page post (photo) code 10, no subcode unchanged: still published and not deletable
The same three posts while still live 200 all stay published / available, not deletable

The Facebook Page post is not detected

For a deleted Page post ({page}_{post}), Graph answers code 10 with no subcode, not 100/33:

HTTP 400
{"error": {"message": "(#10) Object does not exist, cannot be loaded due to missing permission or reviewable feature, or does not support this operation. This endpoint requires the 'pages_read_engagement' permission or the 'Page Public Content Access' feature. ...", "type": "OAuthException", "code": 10}}

MetaAnalyticsResponse files code 10 as permission, so deletedOnNetwork() never runs. With the same token, at the same time:

  • a live post on that Page reads with 200;
  • /{page}/published_posts reads with 200;
  • a made-up post id on that Page ({page}_999999999999999) also answers code 10.

So for Page posts, Graph answers code 10 both when the post is missing and when the permission is. The account has pages_read_engagement. Treating code 10 on a Page post id as "missing" when published_posts reads, the same check you already do for 100/33, should cover it. Facebook Reels use a bare video id and answer 100/33 like Instagram, so they already work.

Checklist

  • Instagram via Facebook: the card shows Deleted, no "View post", and the post can be deleted from the menu (confirm dialog, then gone; GET /posts/{id} returns 404).
  • Instagram Login: not tested; needs an Instagram Login app.
  • Facebook Page post: fails, see above.
  • Facebook Reel: marked deleted; deleted through the API (204, then 404).
  • Threads: not tested.
  • No false positive (partly): the same three posts while still live stay published. I did not revoke a permission or unlink the Instagram account.
  • Story: not tested.
  • Details dialog: the Deleted badge, no link to the network.
  • Member without delete permission: not tested; needs a second member.
  • Duplicate: opens a draft with the same reel, caption and Reel settings.
  • Repurpose: not tested.
  • API: GET /posts/{id} shows status: published, publish_status: deleted; DELETE /posts/{id} works (204), and a post still live is refused with 422 "Sent posts cannot be deleted". MCP not tested.
  • Mobile (390px): the badge fits on one line (82×26 px); no horizontal scroll. The popover is 320 px wide and ends exactly at the right edge of the viewport, with no margin.

The menu on a deleted post shows Duplicate Post, Post Details and Delete Post; Share and Copy link are gone. platform_url is still returned by the API for a deleted post, which looks intended.

Screenshots

Card and the badge's popover:

Card with the Deleted badge

Popover on the badge

Post menu and details dialog:

Post menu

Details dialog

Duplicate:

Duplicate draft

At 390px:

Card and popover at 390px

Paulo Castellano added 2 commits November 4, 2026 12:02
- YouTube uses the statistics it just fetched, so a cache that drops the
  entry can no longer make a live video look deleted.
- Facebook still checks a deleted post on a Page that never granted insights.
- PublishStatus::wentOut() uses match and wentOutCases() a collection; the
  job compares the stored last_seen_at strictly.
- The details dialog computes the rail statuses once and tags the icon.
- Russian badge agrees with the post's gender.
- Tests: the metrics reason on the API and MCP, the lifecycle and PostHog
  activity counting a deleted post, YouTube without a cache, Facebook without
  insights, and the browser test asserts the live link and the closed popover.
- YouTube caches only the statistics it found, so a video missing from
  another video's batch is read again before it is reported deleted.
- One recordRefusal() keeps the refusals and the "object does not exist"
  flag together for every Meta collector.
- The details dialog types its rail statuses instead of casting.
- The PostHog and lifecycle test lives with the PostHog activity tests; the
  parity test builds its posts with count().
- AGENTS.md names every read that stays on live posts only.
Each confirms a deletion from the metrics read it already makes, with an
account-level read (the list its discovery reads) proving the token still
works:

- X: the documented resource-not-found problem for the post, in a 200's
  errors or a 404's body, with no extra request.
- Bluesky: getPosts leaves the post out and its repository answers
  RecordNotFound while listRecords still reads.
- Mastodon: a 404 on an owner read, while the account's statuses still read.
- Pinterest: a 404 on the Pin analytics and on the Pin, while the Pins list
  still reads.

TikTok cannot tell a deleted video from a private one, and LinkedIn, Google
Business, Telegram and Discord have no read after publishing (trypostit#481).
@paulocastellano paulocastellano changed the title Mark posts deleted on Facebook, Instagram and Threads Mark posts deleted on the network as deleted Nov 4, 2026
Paulo Castellano added 2 commits November 4, 2026 12:54
- A deletion check that fails only for now (rate limit, server error) is
  rethrown, so the job retries instead of settling the publication as gone
  without a confirmation (Mastodon and Pinterest lost the detection for good).
- readsMissing() and stillReads() on AbstractPublicationMetricsCollector are
  the one confirmation flow for Meta, Bluesky, Mastodon and Pinterest.
- X flags the 404 in successfulResponse() and decides once in collect(); the
  problem must name this post (resource_id required, also in a 404).
- Bluesky only handles RecordNotFound and leaves the rest to the parent;
  SocialAccount::blueskyService() replaces two copies of the PDS lookup.
- YouTube caches the batch in a plain loop.
- AGENTS.md says which networks need an account read and renders correctly.
- Tests: transient checks retried, X thread root and edge problems, Bluesky
  read once when present and on a self-hosted PDS, Mastodon 410 / 500 / 403
  and the exact authenticated list read, Pinterest Pin refused and the exact
  list read, and the job moving the post for X, Bluesky, Mastodon and
  Pinterest.
- A Facebook Page without insights keeps its permission refusal when the
  deletion is not confirmed, instead of turning malformed.
- readUnlessDeleted() is the one "read, or report the post deleted" flow for
  Meta, X, Mastodon and Pinterest; the confirmation helpers are named
  confirmsDeletion().
- PostFactory::deletedOnNetwork() replaces the repeated published state.
- AGENTS.md: 404 or 410 for Mastodon and Pinterest, and the shared helpers.
- Tests: a temporary failure is retried on every confirming read (Meta post
  and posts reads, Pinterest Pin, Bluesky record), Meta only checks after a
  malformed read and forgets refusals between posts, Bluesky on a fixed
  self-hosted PDS, a publication removed during the check, the copy link and
  the live rail label, a publishing post never shown as deleted, and the
  repurpose Google review label; test helpers deduplicated and the dynamic
  factory call replaced by a dataset of closures.
@paulocastellano

Copy link
Copy Markdown
Contributor

Thanks a lot for testing this on real accounts, that report is really useful, especially the code 10 answer for deleted Page posts. We'll handle it.

I'm testing on my side too and widening the PR: it now covers every network where a deleted post can be detected (X, Bluesky, Mastodon and Pinterest on top of Meta and YouTube), with the rest tracked in #481. It should be ready between today and tomorrow.

@mrbeandev mrbeandev left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please release this , aleast with instagram and facebook support , for now

@mrbeandev
mrbeandev marked this pull request as draft November 8, 2026 16:28
@mrbeandev
mrbeandev marked this pull request as ready for review November 8, 2026 16:28
Paulo Castellano added 5 commits November 8, 2026 20:18
Adds the post.deleted_on_network and post.restored webhook events, guards a
deletion against a discovery that saw the post meanwhile (seen_generation),
confirms a Facebook video refused for permission by reading the video instead
of probing on every read, and writes the network presence transitions in a
transaction so their webhook never reads the post without its timestamps.
Paulo Castellano added 5 commits November 8, 2026 21:30
@paulocastellano paulocastellano changed the title Mark posts deleted on the network as deleted Mark posts deleted on the network, and send webhooks when they go or come back Nov 9, 2026
Paulo Castellano added 6 commits November 8, 2026 22:24
…pdated

The $url closure guarantees a string without a trailing slash, so readers
drop the (string) casts on those keys and the account server accessors no
longer trim. PostObserver::updated returns early when publish_status did not
change and leaves the deleted-origin check to EventType::fromPublishStatus.
The webhook dialog browser test counts the events from EventType instead of
a fixed six.
One enum method returns the Graph API of Facebook, Instagram (both
connections) and Threads, replacing instagramGraphBaseUrl and the abstract
graphBaseUrl/baseUrl hooks of the metrics collectors and the carousel
resolvers. The Instagram and Threads resolvers were the same class with a
different base URL, so they become one MetaCarouselResolver.
data_get() only uses its default when the key is missing, so an account
with "service": null or "instance": null made the accessors return null and
throw; the publishers' earlier ?? covered it. Also names the network in the
deleted-on-network message from Platform::label(), and locks a post's
publications by id only.
The read queued when a discovery restores a post ran as a regular read, so
a post past its metrics window was skipped and a stale discovery could leave
it published for good. It now runs as a baseline read like the daily
dispatch and the page queue do, and refreshes a read already done today.
The Bluesky deletion test reads the account's stored PDS instead of a
literal host.
BlueskyLexicon::xrpc() joins a server and a method, and TelegramApi::botUrl() builds the bot link, so call sites no longer concatenate them by hand.
The /api/v1 prefix lives in one place instead of every call site.
@paulocastellano

Copy link
Copy Markdown
Contributor

please release this , aleast with instagram and facebook support , for now

I will release it today.

Paulo Castellano added 9 commits November 9, 2026 12:20
Graph answers a deleted Page post's own node with trypostit#10, which also means a missing permission, so the deletion was never confirmed. Its insights edge answers 100/33 (checked against a real deleted post).
Insights list deleted publications, but their details answered 404 when no TryPost post was linked. Web, API and MCP now return them with a null permalink, and the dialog shows the deleted-on-network badge.
The four reasons were string literals across ReadPublicationAnalytics; the enum holds them, and the front end types the union.
A 200 with the resource-not-found problem now goes through readUnlessDeleted like a 404, so the lookup no longer carries its errors back to collect().
…place

isDeletedOnNetwork() and postPermalink() are pure helpers, so they live in lib/postPublication.ts instead of the card actions composable. getPostCardStatusConfig() gives the status a post shows, deleted on the network included, which the details rail and the deleted badge now use.
X answers a deleted post with a 200 whose errors name it, single and batch lookups alike (checked against a real deleted post). The lookups check it where the post id is known, so the collector keeps no post id state, needs no successfulResponse override and no readUnlessDeleted wrapper.
The X collector reads the problem type through App\Enums\X\ProblemType and keeps the rule for a deleted post in namesDeletedPost().
Paulo Castellano added 2 commits November 9, 2026 13:20
fromApiResponse() and isConfirmedDeadToken() match the enum cases instead of basename() and string literals; an unknown type keeps its full type URI as the platform error code.
@paulocastellano
paulocastellano merged commit c3b93ee into trypostit:main Nov 9, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Normalize third-party URLs once in config/trypost.php instead of trimming them at every read Remove DEPLOYMENT.md from the repo root

2 participants