probably all the posts in the “fediverse” have been taken already
I don’t doubt this, as it’s trivial to do (by design). You don’t even have to scrape anything.
New posts and comments in a Lemmy community (or Mastodon profile or any other ActivityPub-based Fediverse app) get pushed to every server where federation is allowed and at least one person is subscribed to it. You can spin up your own server, subscribe to a bunch of communities, wait a while, and over time your PostgreSQL database will be populated with profiles, posts, comments, etc.
I think it’s safe to assume that any public code on the internet is already in their training dataset.
At this point anything that is public in general, probably all the posts in the “fediverse” have been taken already
I don’t doubt this, as it’s trivial to do (by design). You don’t even have to scrape anything.
New posts and comments in a Lemmy community (or Mastodon profile or any other ActivityPub-based Fediverse app) get pushed to every server where federation is allowed and at least one person is subscribed to it. You can spin up your own server, subscribe to a bunch of communities, wait a while, and over time your PostgreSQL database will be populated with profiles, posts, comments, etc.