So I used to think the folks at SourceForge were just overworked and under-appreciated. They've worked really hard recently to show me this is not the case. Sure, it's a "free" service, but it's there to sell their premium services, and they get a lot of exposure being the place to go for open-source development.
They've been having growing pains for a while; a few months ago, they had a major CVS outage. Open-source development the world around ground to a standstill for days while they worked to get hardware up. Since then, they've been planning on transitioning folks to subversion, and to a new CVS infrastructure. Did I mention that in the meantime, the anonymous CVS has been frozen and out-of-date since March? This outage was only "repaired" for developers; user access has been broken all this time.
Since then, we've been planning on moving to our own server (donated by xs4all). As you've seen in previous posts, I've made progress towards that end. In the meantime, our expectation was to get in on the "new CVS" beta. We've been preparing versions of Fink that can "phone home" to figure out what the CVS repository should be, so we'd be prepared for when the switch came.
2 days ago, another CVS outage occurred. After a day of waiting without the site status page being updated, I opened 1 of (I think) hundreds of open unanswered bugs about CVS being down, making note that they should at least update the site status, so people would stop opening bugs. I went to the #sourceforge channel on IRC and asked about it, and they said there were network driver problems. They got it fixed, and CVS was back up, for, literally, 30 seconds. It turns out that machine also had a hard drive out. After being told it would be up once the hard drive was replaced, I wandered off to nap some more. I'd been out sick for 2 days and was really hoping to commit the culmination of 2 weeks of work on KDE fixes for outstanding issues that were (and are) affecting nearly all Fink KDE users. I had made the last few changes while I was at home, sick, and could finally commit and get back to sleep.
Hours pass, and I finally ask, "I hate to keep bugging you guys, but it seems the site status never gets updated during outages like this. Were there problems putting in new drives? Is there something else keeping our cvs from coming back up?" The response: "I'll update site status tomorrow with details, I don't have full details at the moment, but I do know it will be down over night, at minimum." Understandable, things happen. But the fact that the site status page wasn't the first thing updated as soon as a problem was found shows me how much they care about their users. You can save a lot of ill will by just telling people what's going on.
Today, the site status gets updated:
( 2006-05-10 04:43:14 - Project CVS Service )
As of 2006-05-09 the developer CVS
server had a disk-failure. As the new CVS infrastructure is in
its final phases of rollout, we'll be deploying it, in place of
the current infrastructure, by end of week. We'll be sending
out an email to project administrators with further details
later in the day, regarding how to access the new CVS servers
and the changes that occurred with the new infrastructure.
So one thing to note. Anonymous CVS for the new servers is at a new CVSROOT. Everyone that wants to access the new CVS repository will need to check out fresh, or manually munge their CVS/Root files. This is the eventuality we were going to handle gracefully, which is now impossible.
Additionally, shell access to the project servers (ie, the way we log in to modify pages on http://fink.sourceforge.net/) is down, so basically:
- we can't get to CVS
- when we do, our users won't be able to get to CVS
- we can't update the old CVS to give users a version of fink that can find the new CVS
- we can't update the web site to tell people what's going on
- once we can, the only option we have for our users is going to require manual intervention of some form, at the very least switching to
selfupdate-rsync
This is intensely frustrating. It would be one thing if it were just hardware and server issues. These things happen. But the total lack of disaster planning and managing of user resources (and user expectations) is just abysmal. The timing makes it all that much worse, since we were literally prepared to finally move away from SourceForge's spotty service in the next few weeks (or, at the very least, months).
But don't worry, SourceForge, we'll be reducing your server load just as soon as we're able. You can count on a little less load in the future.
Share on Facebook
A *little* less load? I’d guess it’s more like 5% load :-p
Thanks for the update
/me switches to rsync
what a mess 🙁
It’s for reasons like this that I left sourceforge some months ago and set up my own site at http://source.emptycrate.com.
Agreed, I have experienced many difficulties aswell. I am not one to point fingers at anything in particular, but SourceForge to me has not been a reliable service. Good Luck! With everything.
Bad news, lets hope they fix things quite quick !
Doesnt sound too good at all :\
I really appreciate the free service I’m getting from SF, but we’ve come to a point where I need to reevaluate my options. Their reliability problems are slowing our project down, in particular with regards to CVS.
What are good alternatives for open source projects? I don’t want to do nongnu.org / Savannah because my project runs on Macs and I (thus) have problems with their “free software” ideology. BerliOS?
SourceForge’s CVS went belly-up right before I was going to release MySQLdb-1.2.1. Fortunately, I was able to use their utility to convert to Subversion, which had become availble a few weeks earlier. I was going to convert *after* the release, but the outage forced the issue. So far I haven’t had any issues with their SVN service.
However, their compile farm is kind of a joke. The Mac OS X servers have been down for months, and the other available servers/OS are mostly pretty old.
I’ve also never used their website service because of the restrictions on it. I put some HTML documentation up there, because their Doc system does some really unpleasant things with stylesheets and makes my HTML-from-ReStructuredText look really ugly, but then I found that I couldn’t put ads there, so I scrapped the entire idea; it wasn’t worth the trouble.
The “Quick Release System” has *never* worked for me: the file upload always fails.
Generally, with the exception of the CVS outage, it’s been reliable for me, but there’s a lot of really clunky stuff and other annoyances.
The http://wicketframework.org project was fortunate enough to move to subversion in februari just before the major outage. Though subversion support in Eclipse isn’t too great, and the initial connection is slow, we are very glad that we switched.
Sofar, our experience with subversion is great. Until the next major outage…
In the meantime, the SVN service is really nice, but this whole situation makes me wonder if they’ve done any better redundancy planning for that service.
Redundancy, clustering, and farms guys!
As the one who is usually responsible for PR/Fundraising work and as a board member of the FDN I can only congratulate RangerRick for writing this up. The situation is unfortunate and while we realise that this is a free service, huge projects like Fink bring a lot of hits to VA and thus ad revenue. As such there such be a mutual benefit from this relationship.
I am sure that some might have made offers to help already, I just wanted to assure you all with this comment, that we are very welcoming to such offers. We are not afraind of sponsorship agreements and we have worked hard to get FDN registered as a not-for-profit.
Thank you, to all of you who use Fink and believe in us.
You might take a look at Freepository (https://freepository.com). Free secure CVS hosting (up to 300mb for free), sserver connection (supported by the 8 or 9 most widely used CVS clients), and very unobtrusive terms and conditions. Freepository has over 14000 members and hosts over 10000 projects, and was created exactly to address the concerns many people express over Sourceforge. There is no requirements at placed on your project (i.e. licensing or other). There is a preimum account that provides defect tracking and a private wiki for $6 a month.
Thanks,
Bryant
https://freepository.com
As a user, not a developer – i think you’re missing one thing. Sourceforge IS the place to go. If I’m looking for something that’s opensource that solves a problem I have, I know that I have somewhere to turn.
I can imagine that as a developer sourceforge sounds like a nightmare, but please spare a thought for the users who rely on central locations like sourceforge to even show us your projects exist in the first place.
While managing a project on SF (http://www.tendersystem.com) for slightly over a year we experienced the same CVS issues.
We setup our own development CVS and only do CVS commits to SF every now and then.
This has worked well for us as all developers have access to a working CVS and pre-release testers get their source from SF.
I do not know if it will work for Fink as it is a much smaller project.
As an informed tech-savy user, Google is the place to go to find open source software that solves your problems.
SourceForge might be a major host in the open source world, but there are better ways to find good open source software than browsing or searching solely SF.
I’m a Mac/UNIX developer who uses SourceForge heavily. When developer CVS went down, my project (Open Babel) had already had brief discussions about moving to SVN.
Oh, the headaches we had — because the anonymous CVS server at SF was out-of-date with the developer CVS, we had to wait for someone at SF to make a CVS tar for us. Then I had to download, do some filtering (we had a case-sensitivity issue), convert to SVN myself and upload again.
Now, things look pretty good.
But I feel your pain. SF seems to be having really bad scaling problems right now. The worst seems to be a personnel issue — they just don’t have enough manpower to keep up with the problems. (Compile Farm, CVS, Mailing lists, statistics…)
Moving away from CVS might be in Fink’s best interest, regardless of SourceForge headaches.
At OpenNMS, we used to host everything – website, mail, CVS, etc., on our site, although we did use Sourceforge in the beginning (we’re project 4141). So while we had 110GB downloaded every month, our Sourceforge activity rating was like 47%.
Now we moved CVS and mailing lists back to Sourceforge, and we are usually in the 300-400 most active projects with a 99+ % rating. We luckily made the SVN jump just in time, and I am usually happy with the mailing list service (much less admin overhead dealing with things like spam) but the important thing is that someone looking for open source network management sees our project as being alive instead of dead. To paraphrase Oscar Wilde, the only thing worst than being on Sourceforge is not being on Sourceforge.
It\’s all going downhill now that Colin is gone. He was the true visionary…
Thanks for the updates Rick. Sounds pretty frustrating. Did microsoft buy them out or something? I’ve long since moved my projects off of sourceforge as well for similar reasons. They seem to be really taking the stand that the people purchasing premium accounts get good service, the free users get to be beta testers and use all the old broken stuff more often than not.
I’m about to head for the door at sourceforge myself. Like many others here, I have a reasonably popular project there, but the recurring theme of downtimes lasting a month or more is wearing thin.
Anoncvs has been dead for over a month. Developer cvs has been dead most of this week. At this moment, five of the hosts on compilefarm are dead and the ones that aren’t dead are so far out of date that it’s hard to build anything on them. Web reliability is poor. Convincing them of problems is essentially impossible.
It’s a definite trend and I”m tiring of it.
May I suggest you look at http://www.launchpad.net? They’re growing quickly, they’re financially stable, and they’re not seeking to upgrade anyone to paying accounts. Just a thought.
Hello Ben,
We are sorry to hear about your recent problems.
We at ibiblio would be happy to host your project.
ibiblio is one of the oldest and largest repositories of free & open source software and information. We host/mirror a number of high profile software projects and linux distributions including the linux documentation project, torrent hosting for eclipse, gutenberg, groklaw, etree, gnu, etc. And we would be glad to add your project to the list.
We would like to discuss how we can help your project including hosting and/or services. We starting to roll out SVN and Trac services too.
ibiblio is based at the University of North Carolina at Chapel Hill. We have excellent infrastructure with large server cluster farm with access to Internet2, National Lambda Rail, and top-tier commercial networks.
Please contact us ( help AT ibiblio DOT org )with your requirements and any other details/questions.
Thanks,
Sayan
We came to this conclusion at phorum.org several years ago. We had our own web space but wanted to be part of the SF buzz. We used their CVS and used them for downloads.
After several things went wrong in CVS and we grew tired of the horrible interface for uploading files to their server, we just quit it.
We are much happier now running our own SVN and mirroring stuff as needed ourselves.
Just a quick note to ‘sffan’ .. Colin Bodell didn’t have anything to do w/ SF.net in any official capacity… he was the CTO for VA (not OSTG) in charge of the commercial version of SourceForge. I reported directly to the man. SF.net’s issues are of their own making, not due to the leaving of a C-level officer of their parent organization
Just a quick note to Marius.. Don\’t know what you\’re talking about, but it smells fishy. Just look at teh source forge blog to see that Colin was majorly involved http://blog.dev.sf.net/index.php?/archives/P3.html
\”About a year ago I began to spend more time on SourceForge.net, and the other OSTG properties. Now I\’m dividing my time about 50/50 between SFEE and SF.net.
I\’m also looking after product management for the next generation of SourceForge.net.\”
With posts from him quite often after that. The site was stagnant for years until he got involved.
Now he left the company and the site is sucking. That\’s okay I am sure it will not suck for those that start paying money for it. Or the people who pay for support from the projects you host.
{I know it\’s hard when the big players at a company leave. But it only makes things fishier to explain that they were not really involved.}
Thoughts about the CodePlex Beta
Imagine a blog entry where I talk about my first thoughts of CodePlex Beta that went live yesterday.