We’re aware of ongoing federation issues for activities being sent to us by lemmy.ml.

We’re currently working on the issue, but we don’t have an ETA right now.

Cloudflare is reporting 520 - Origin Error when lemmy.ml is trying to send us activities, but the requests don’t seem to properly arrive on our proxy server. This is working fine for federation with all other instances so far, but we have seen a few more requests not related to activity sending that seem to occasionally report the same error.

Right now we’re about 1.25 days behind lemmy.ml.

You can still manually resolve posts in lemmy.ml communities or comments by lemmy.ml users in our communities to make them show up here without waiting for federation, but this obviously is not something that will replace regular federation.

We’ll update this post when there is any new information available.

Update 2024-11-19 17:19 UTC:

Federation is resumed and we’re down to less than 5 hours lag, the remainder should be caught up soon.

The root cause is still not identified unfortunately.

Update 2024-11-23 00:24 UTC:

We’ve explored several different approaches to identify and/or mitigate the issue, which included replacing our primary load balancer with a new VM, updating HAproxy from the latest version packaged in Ubuntu 24.04 LTS to the latest upstream version, finding and removing a configuration option that may have prevented logging of certain errors, but we still haven’t really made any progress other than ruling out various potential issues.

We’re currently waiting for lemmy.ml admins to be available to reset federation failures at a time when we can start capturing some traffic to get more insights on the traffic that is hitting our load balancer, as the problem seems to be either between Cloudflare and our load balancer, or within the load balancer itself. Due to real life time constraints, we weren’t able to find a suitable time this evening, we expect to be able to continue with this tomorrow during the day.

As of this update we’re about 2.37 days behind lemmy.ml.

We are still not aware of similar issues on other instances.

Update 2024-11-25 12:29 UTC:

We have identified the underlying issue, where a backport for a bugfix resulting in crashes in certain circumstances was accidentally reverted when another backport was applied. We have applied this patch again and we’re receiving activities from lemmy.ml again. It may take an hour or so to catch up, but this time we should reliably be getting there again. We’re currently 4.77 days behind.

We still don’t have an explanation why the logs were missing in HAproxy after going through Cloudflare, but this shouldn’t cause any further federation issues.

Update 2024-11-25 14:31 UTC:

Federation has fully caught up again.

        • Psythik@lemmy.world
          0·
          3 days ago

          Because they’re tankies, and I really don’t need to explain to you why that’s a bad thing. You already know.

            • givesomefucks@lemmy.worldEnglish
              01·
              1 day ago

              A tankie is someone who sided with the authoritian government and their tanks against the one guy who stood in front of them in Tianenman Square.

              They say they’re communists but defend an authoritarian oligarchy…

              Authoritarians say they’re communist because they hate communism and don’t want people to think it’s a good idea. So they call everyone on the left a tankie.

              It’s a stupid label and 99% of the people throwing it around don’t understand anything about it and are just lying about what communism is.

              • Maalus@lemmy.world
                1·
                1 day ago

                No, being a tankie isn’t only about supporting aggression / silencing people, it is specifically about romanticizing communism as some great philosophy that solves every problem and in extension sucking the D of Russia and China, while going “every other country bad” to the point of complete insanity.

                Otherwise rightwing US are tankies.

    • Hamartia@lemmy.world
      0·
      3 days ago

      The hypocracy of people exaggerating the scale of the problems and piling-on about other instances in their manic drive to turn this place into an echo chamber is depressing.

      • GeneralInterest@lemmy.worldEnglish
        0·
        2 days ago

        “Echo chamber” is often used as a criticism on the internet. But in real life, when you make friends, it’s with like-minded people, right? So maybe it makes sense to seek online spaces with like-minded people, rather than spaces with people of various views attacking and insulting each other.

        • Hamartia@lemmy.world
          0·
          2 days ago

          It’s a reasonable argument for giving you tools for tailoring your own feed but not for censoring entire groups out.

            • Hamartia@lemmy.world
              01·
              1 day ago

              It makes sense.

              Not everyone on .ml wants to eat your dog. Just as not everyone on .world is Trolleyman.

              I’d say that the majority of people on the fediverse are capable of having a civil conversation on most topics. Some might have a handful issues that they dig their heels in over but can be civil if you comport yourself respectfully. And then there are a tiny minority of narcissistic zealot idiologues that try to maintain a narrow normative pressure on permissable opinions.

              • A_Random_Idiot@lemmy.worldEnglish
                1·
                1 day ago

                Remember folks.

                This is how they infiltrate circles and twist them to right wing authoritarianism and extremism

                They send in someone who uses a soft approach as their opening gambit, talking about how they are good people, and how they don’t deserve to be excluded, and its really unfair to judge them for what they and their associates do, and that if you were really as tolerant and as leftist as you said you were you’d accept them, and engage with them, and sup with them.

                While handwaving any of the actual, legitimate, pointed criticisms as the actions of a “tiny minority of zealots”

                Its death by a thousand cuts, and what Hamartia here is doing is trying to do the first tiny cut. To try to make you feel unreasonable, and possibly even bigoted, for taking a stand againts the poor, innocent nazis/tankies/etc.

                You know why this hasnt been a significant problem until this past decade? Cause these people were punched in the mouth the second they started "Well, Ackshually~"ing in the defense of right wing fascist extremism. The second they started to fly their freak flags. Its why they have a whole underculture of co-opted symbology and secret coded language, because they were scared and had to hide, and needed a way to recognize eachother without risking a baseball bat to their jaw.

                And the reason we stopped doing that, is because this soft handed approach of theirs worked. Its why they dropped the coded language. Its why their freak flags are flying high and proud now. Its why they are openly, gleefully saying their absolute horrid, unfiltered hated with gleeful abandon now. Because what Hamartia is doing, worked. It got their foot in the door, and as soon as it was there, they crushed in with the entire party and took over.

                You can see where we are today, and its because we let their faux-reason, concern trolling, and paly acting exploit our tolerance.

                And you see what its gotten us

                And you see why its better to go back to the old classics of punching these motherfuckers in the mouth.

                So don’t let this soft hand propaganda being done by Hamartia and others sway you. Don’t let it make you feel unreasonable, or uncivil. They are manipulating you to open the door for the ones behind them that arent capable of feigning and playacting “polite” discourse in attempt to make you feel like you’re the bad guy.

                These people have no place in a polite and tolerate society. They are a cancer that seeks to destroy it, by exploiting our beliefs against ourselves. Its not paradoxical. Its not hypocritical. Its not whatever bullshit attempt they make next at trying to twist tolerance into a dagger for them to use to stab tolerance itself in the back and kill it to their extreme benefit, and to societies great loss.