Ruby, Rails and RSpec burned a day off my lifespan
The big project we’re working on needs to be demo’d to the board next week. It won’t be finished but with only 3 weeks to go before we beta it’s important we show good progress. Understandably then the entire team is under the gun and working really hard to get the demo ready.
I’ve got a ton of code of my own to do for this and am working nights, mornings (from about 4am) and weekends to get my own work done in and around my duties as the team lead. So naturally, when I started to make some really good progress yesterday, that was the point in time that the 3 amigos (Ruby, Rails and RSpec) decided to go a little obscure on me.
I needed to rename a model, controller and views. Subversion decided to be a pain in the ass and throw a fit at so many svn mv commands being issued which threw me for a loop for about an hour trying to fix everything subversion wise. When I did get it fixed, I ran ‘rake spec’ to test the project and found a ton of failures. Larry had come over by this point to complain loudly that I’d broken the source. I made some changes, and checked it back in in the hopes of quelling some of the errors by fixing the obvious. I still had a lot of failures though.
As I started to look around the errors though, none of them made any sense. We’re not just talking about the usual programmer beating their head on an obvious wall kinda weirdness, but real freaky ships vanishing in the Bermuda Triangle weirdness.
Dray took a look, grabbed the source and ran ‘rake spec’ himself. No errors. Larry did the same. No errors. I did it again - a ton of errors, but nothing I spent about 2 hours working on this then gave up and came home only to carry on again last night.
No matter where I looked, and what I tried, nothing would make the errors go away. I put the project on another Mac at home and got zero errors, but my work macbook kept showing me a flood of them.
I got up this morning at about 3:30am and started again. I decided to go back to basics. I compared the Ruby gems on my Macbook with those on my Mac Pro and stripped out anything different. I uninstalled gems with different versions and then reinstalled them with the same versions as on my Mac Pro. Still errors, and still the Mac Pro told me nothing was wrong.
I built my development environment from source, so in desperation I re-installed Ruby. Still I had errors. I zipped up the Ruby install and the gem library on my Mac Pro and replaced the one on the Macbook with it. Yup, the errors were still there. Throughout this all I should point out that running an individual test suite on both machines never gave me any errors at all.
Completely desperate, after about 8 hours on this, I started to fire the command Rake Spec was issuing but removing tests one by one to identify just what was going on. Eventually I found 4 tests that were all calling Ruby’s Lambda method. It just looked weird to me so I commented out those lines in each of the 4 tests. I ran rake spec and everything passed.
Gingerly I started to put the lambda calls back in, one by one, until eventually they were all back the way they should be. I have no errors now.
But, I didn’t do anything! If I grab the source tree from subversion onto my MacBOok now, something I’ve done countless times in the past day to see if the problem would magically go away, the problem seems to have magically gone away. I have no idea why.
Sometimes Ruby and Rails are just obscure that I want to permanently can them and switch to Python and Django forever.
Honestly, other than the speed boost that python gives you, you are no better off with Django than you are with Rails. It has it’s many, many, many, MANY quirks. Not advocating rails but if we’re going to nitpick frameworks here…
Comment by Trevor — October 5, 2007 @ 5:48 pm
I disagree.
The big argument in favor of Rails is productivity. It is a framework that gains traction based on the productivity gains teams experience when they adopt it. It does a whole lot for you underneath the covers that you would normally have to hand code in any other framework, and so the argument goes that since most developer team’s are expensive compared to the cost of boxes to run the software, you should spend money on boxes (to scale) and save money in terms of cost of time to market.
The problem with the argument though is that scaling is a bigger story than most advocates think. Even DHH is dismissive when it comes to scalability but it’s a well known fact that 37 Signal’s own work (Campfire most notably) relies on C extensions to be highly performant. DHH simply say that the scalability argument is annoying, and moot, but I’ve yet to see definitive guidelines for how to deploy a Rails application of any complexity to handle upwards of 500,000 users. Twitter hit a wall with performance and their app is very simple. We hit the same wall, with a much smaller userbase. When that wall gets hit the costs go up exponentially. Suddenly you’re looking at an entire team trying every trick in the book to eek out more performance, and you’re looking at that lost productivity cost in addition to the cost of yet more boxes, racks, etc etc.
That is all to say, of course, that the speed boost of Python shouldn’t be discounted. But Django, as a framework, is highly productive too. It also sports a much more robust data mapper than Active Record, a much cleaner MVC model, and a lot less “magic”. I’ve seen more hours of lost productivity due to programmers getting confused by the Rails magic than I care to recount here. I’ve also seen a lot of lost hours as programmers struggle to climb the Rails magic infused learning curve. Django, by comparison, is simple. So is PHP Cake. So is Grails for Java. Suddenly the Rails productivity argument starts to look a little weak. If I can get programmers every bit as productive with Django as I can with Rails, and save a lot of money on server costs because Python is faster, and save a lot of money in terms of learning curves since Python code is, I think, more readable, why would I go with Ruby on Rails? Why do I allow my team to continue with Ruby on Rails?
At this stage in the game the answer to that is two fold; team and legacy. We have climbed the learning curves and are proficient in Rails now. It would cost more to learn a new technology. Also a high percentage of the team work at the company because we are a Rails shop. We’d lose those people, and the knowledge and experience they have gained, by switching. I do see changes on the horizon for both the Ruby runtime and for Rails that seriously address performance and we’re in a safe enough position right now to be able to hold out for those visions to become real.
I still think Python and Django would have been a better choice from the get go though.
Comment by pwrighta — October 5, 2007 @ 6:48 pm
Cant go wrong with Django. Switching to Python & Django was one of the best decisions I’ve ever made.
Comment by Matt — October 15, 2007 @ 9:35 am
“I built my development environment from source, so in desperation I re-installed Ruby. Still I had errors.”
Not exactly sure what you mean there. I always use ruby from source.
I also avoid any gem-only release - it shows the author is clueless and believes the world exists only as gem-package.
What error exactly have you been struggling with? It might serve as a role model for these authors who believe that gems are the panacea.
Comment by aye — October 15, 2007 @ 10:10 am
Sounds like subversion was screwing you. Should have just checked out a clean copy and gotten back to work.
Comment by ryan — October 15, 2007 @ 10:58 am
You made changes in subversion before you ran the tests locally? Are you mad?
Comment by Asd — October 15, 2007 @ 11:00 am
Interesting. Wouldn’t say that I am ‘clueless’ though aye.
I had the source that I’d built everything (ruby, imagemagick etc etc) from and re-built and re-installed all the tools after cleaning the binaries out.
It’s looking more and more like the problem was actually related to subversion doing something weird, and yes, asd, I goofed up by not running my tests before check in.
As for ‘what error’ I had aye, if you read the post you’ll see no error. It was random, completely weird, and totally not obvious test failures that only happened on just one machine, and no other.
Comment by pwrighta — October 15, 2007 @ 12:34 pm
It’s unfortunate when these sorts of headaches come up, but they will come up. I’ve seen them at least once with every team I’ve been on, and they show up no matter what tools are being used.
I would chalk it up to team dynamics in part — every team will have some struggles learning to work together and learning to work with the toolset they’ve chosen to use. There will almost always be someone who is new to the team or the tools or good methodologies in general (i.e., even if you’ve gotten rock-solid someone else won’t be). Usually the toolset is a new toolset, at least in part (how long has the language been around, how about the frameworks, how about the version control system, how about the server/container/UI software, how about the testing tools, continuous integration?, etc.). Second projects using the same toolchain with the same team tend to go a lot better.
The only thing I can recommend is moving one step at a time — that is, changing only one variable at a time. So, if the tests aren’t running, don’t recompile the language interpreter. It can be very difficult to isolate version control problems like it appears that you had. Sometimes doing a fresh pull from trunk in a parallel directory and generating a patch with svn diff and applying it will work. Sometimes svn export from the working directory to a new directory, etc. But, there’s no easy solution, just step back, take a walk (or whatever gets you in a clear state of mind), and come back and try the simplest thing that could help. Try to build up a model of what’s happening and make non-destructive experiments to refine that model to help solve the problem.
The other thing, which I’m sure you’re doing right now, is thinking about how you can avoid the problem, or similar problems, in the future. Sometimes it’s as straightforward as saying “if I’m going to do mass-renames, I need to get on an SVN branch before it starts so I don’t trash the tree”. With mercurial or git this might not be necessary. With CVS it might not be possible at all.
I tend to recommend doing small commits to version control, frequently, with test runs before every commit. This means your tests need to run fast, and ideally you need good test coverage. That might help the general problem you had (lots of stuff out, i.e., lots of variables changed, and hard to isolate which one was critical). Small changes, committed frequently, with tests run continually Autotest is great for this with Ruby projects. TDD or BDD with Autotest helps even more.
I’ve also been on the “3:30am shit is way broken I need this tomorrow” knife edge. There are two things that help this, to my mind: (1) be more Agile: have fewer (== 0) arbitrary deadlines, measure progress on iterations, and give frequent views into the state of the system for everyone (both developers and customers); and (2) spread the knowledge around: be pairing, have more developers who can work through the code, more folks you can hand off to, when you make your small commit the next folks can run with it.
This stuff is hard, software development is hard, but mostly being successful is about being open, transparent, honest, collaborative, and disciplined about good techniques (of which Agile techniques tend to overlap quite frequently). All of that is mostly independent of what tools you’re using, though the tools can eliminate some headaches if your practices are good.
Best of luck,
Rick
Comment by Rick Bradley — October 17, 2007 @ 10:05 am
Hmm I think its a mac file system caching thing. I seem to recall encountering this 3 or 4 times since I’ve been using my macbook for 2 years.
Basically the files in my folder doesn’t register as the actual / updated files that they should be (almost like its cached and the cached version doesn’t get stale)
Anyway that is weird. As for python ruby argument, I think most of the reason why people are into rails now is because it came a lot earlier than django etc, and by the time the web scene for python was good enough, people were hooked into rails. just my 2c. (Personally I think Ruby is just slightly better than python)
Comment by Cyril — October 17, 2007 @ 10:19 am
[...] DHH, such as Imploding Rails, Jesus DHH, and the Uncle Ben Principle, 10 Areas Where Rails Fails, Ruby, Rails and RSpec burned a day off my lifespan, and How to ruin a Rails [...]
Pingback by Juixe TechKnow » Ruby on Rails Radioactive Fallout — January 4, 2008 @ 1:01 am