Hey from the GitHub team. Outages like this are incredibly painful and we'll share a post-mortem once our investigation is complete.
It stings to have this happen as we're putting a lot of effort specifically into the core product, growing teams like Actions and increasing performance-focused initiatives on key areas like pull requests where we're already making solid progress[1]. Would love if you would reach out to me in DM around the perf issues you mentioned with diffs.
There's a lot of architecture, scaling, and performance work that we're prioritizing as we work to meet the growing code demand.
We're still investigating today's outage and we'll share a write up on our status page, and in our February Availability Report, with details on root cause and steps we're taking to mitigate moving forward.
Literally everyone who has used Github to look at a pull request in say the last year has experienced the ridiculous performance issues. It's a constant laughing point on HN at this point. There is no way you don't know this. Inviting to take this to a private channel, along with the rest of your comment really, is simply standard corporate PR.
Yes agreed it's been a huge problem, and we shipped changes last week to address some of the gnarly p99 interactions. It doesn't fix everything and large PRs have a lot of room to be faster. It's still good to know where some worst performance issues are to see if there's anything particularly problematic or if a future change will help.
FWIW, I find the new React-based diff viewer worse than the old server-rendered page. I disabled the preview for this reason. It does have some nice features but overall it feels more finicky. I would think that in theory this should be better at handling large diffs but I'm not sure that that's the case, and at least the UX feels more choppy.
That's financialization at play. When you render and syntax highlight the diff on the server, Github pays the cost, if you do it on the client side, the cost is paid by the client. At Github's scale it's probably a large enough of a difference that they decided the reduced customer experience is worth it.
I have been using GitHub since 2011 and it's undeniable that the performance of the website have been getting worse. The new features that are constantly being added are certainly a factor, but I think the switch to client-side rendering that obviously shifted the load from their server to our browsers and also tend to produce ridiculously large and inefficient DOMs[1] is the main cause.
If you want a practical example, here you go. I'm a Nixpkgs commiter and every time I make a pull request that backports some change to the stable branch, GitHub unprompted starts comparing my PR against master. If I'm not fast enough to switch the target branch within a couple of seconds it literally freezes the browser tab and I may have to force quit it. Yes, the diff is large, but this is not acceptable, and more importantly, it didn't happen a few years ago.
It's insulting to see the word "progress" being used when the PR experience is orders of magnitude slower than it was years ago, when everyone had way worse computers. I have a maxed M5 MacBook and sometimes I can barely review some PRs.
Hopefully the published postmortem will announce that all features will be frozen for the foreseeable future and every last employee will be focused on reliability and uptime?
I don’t think GitHub cares about reliability if it does anything less than that.
I know people have other problems with Google, but they do actually have incredibly high uptime. This policy was frequently applied to entire orgs or divisions of the company if they had one outage too many.
For what it's worth, I doubt that people think it's the engineering teams that are the problem; it feels as though leadership just doesn't give a crap about it, because, after all, if you have a captive audience you can do whatever you want.
(See also: Windows, Internet Explorer, ActiveX, etc. for how that turned out)
It's great that you're working on improving the product, but the (maybe cynical) view that I've heard more than anything is that when faced with the choice of improving the core product that everyone wants and needs or adding functionality to the core product that no one wants or needs and which is actively making the product worse (e.g. PR slop), management is too focused on the latter.
What GitHub needs is a leader who is willing and able to say no to the forces enshittifying the product with crap like Copilot, but GitHub has become a subsidiary of Copilot instead and that doesn't bode well.
>I doubt that people think it's the engineering teams that are the problem
Did you forget Microsoft engineering response to Casey Muratori "Extremely slow performance when processing virtual terminal sequences"?
"I believe what you’re doing is describing something that might be considered an entire doctoral research project in performant terminal emulation as “extremely simple” somewhat combatively."
We'll ship some initial changes here next week to provide maintainers the ability to configure PR access as discussed above.
After that ships we'll continue doing a lot of rapid exploration given there's still a lot of ways to improve here. We also just shipped some issues related features here like comment pinning and +1 comment steering [1] to help cut through some noise.
Interested though to see what else emerges like this in the community, I expect we'll see continued experimentation and that's good for OSS.
Hey I'm on the team at Retool and this is a place we're spending a lot of time right now. We're still working through the challenge of showing the depth of the IDE while balancing the goal of a lot of new users to maybe just get an admin panel or support dashboard that connects to their existing db or salesforce.
I'd love to hear from you (or anyone else reading this) on how your getting started experience went, particularly if you found yourself wanting to just eject into writing a script or app elsewhere. I love to pair on apps whether it's moving existing code into Retool or starting fresh to see the hurdles. My email is my HN username @retool.com
BTW just sent a twitter DM, thanks again for the feedback.
I recently discovered this combination and my mind was blown a bit.
The key part to me is ChatGPT --> Code --> Execute in real world (not sandboxed) --> achieve whatever you want out there via APIs.
I am eager to learn about how this works and maybe look to replicate or push for similar connectivity with other code execution / notebook environments (say Amazon Sagemaker)
If anyone can link me to disccussion groups, active communities or github projects that are toying with these ideas -- I would love to dive in.
@thejosh I'm a PM on Azure Pipelines and I'm interested to hear the builds UI ordering and other wonkiness feedback if you want to share. You can email me at my firstname.lastname at microsoft.com and we can do whatever works from there.
Azure Pipelines PM here. Definitely hear you on the current YAML experience. We're working on updates to the editor now to allow you to quickly browse and insert task snippets. It's going to be a big improvement.
I created a visualization on a similar topic that looked at mortality rates state-by-state using the 2010 census data. It was on HN about six months ago.
According to the Harvard Injury Control Research Center, there is some risk, particularly when considering younger demographics.
"The preponderance of current evidence indicates that gun availability is a risk factor for youth suicide in the United States. The evidence that gun availability increases the suicide rates of adults is credible, but is currently less compelling." [1]
There's a lot of good additional data in the original report. According to the table on page 40 [2], self-harm by firearm represented 19,392 of 38,364 suicides. This dwarfed the number of gun-related homicides which was 11,078.
Page 83 explores death by intent. Interestingly, there's also a category for legal intervention/war which represented 344 of all firearm deaths.
On page 110 of the CDC's PDF the formula for computing age-adjustment is discussed. You'll also find the original data there where you can look column-by-column and see the impact that adjustment has for various states and cases.
It stings to have this happen as we're putting a lot of effort specifically into the core product, growing teams like Actions and increasing performance-focused initiatives on key areas like pull requests where we're already making solid progress[1]. Would love if you would reach out to me in DM around the perf issues you mentioned with diffs.
There's a lot of architecture, scaling, and performance work that we're prioritizing as we work to meet the growing code demand.
We're still investigating today's outage and we'll share a write up on our status page, and in our February Availability Report, with details on root cause and steps we're taking to mitigate moving forward.
[1] https://x.com/matthewisabel/status/2019811220598280410