Developer *Fail* Moments

by - 2012/08/27 24 Comments Development, Random

Each OpenMovie project we manage to get ourselves into some fairly awkward technical difficulties – and even with the best intentions things backfire and break in strange and hard to foresee ways.

Being an open project at least we don’t have to pretend to be `professional` and can share some of the low-lights of the project as we did for Big Buck Bunny and Sintel :)

 

In no particular order…

  • Once NFS became mad on the server Sergey just rebooted the server. ll file systems were forced to be checked at boot time. Checking both 5TB and 3TB RAID took several hours. No internet and no production SVN meanwhile.

 

  • Sergey made some tweaks to file system layouts on all systems and made a typo in file system mounting rules. Next day nobody was able even to boot their computers.

  • Campbell decides to give someone else the fun of fixing broken (but awesome) 8 core XEON, we get it back from the shop `fixed` with only 2 memory slots limiting the ram making it fail at rendering the majority of our scenes.

 

  • For some reason Samba server (server to allow mac and window users access our shared folder) was misconfigured and files with wrong permissions were creating. Trying to fix this Sergey used a mask of “.*” to change permissions for all files including hidden one. Who knew that this mask would also match “..” folder which is parent folder. Ended up with all files granted to full access to anybody on the server.

 

  • In the middle of the project it was discovered a typo in automated Ubuntu installation script — it created 25GB partition for data and couple of hundreds of gigabytes for swap.

  • Campbell wanted to download all versions of a file and whipped up a clever script to grab the history of all versions at once,
    …turns out that making 120 connections to blenders svn server is enough to blacklist the blender institute from connecting to our source code repository, now artists are blocked from blender updates and devs can’t apply changes to blender. Resolved the next day by changing our routers MAC address to get a new IP.

 

  • Sergey powers up all remote DELL renderfarm nodes and manages to overload their PDU (Power Distribution Unit) 4 days before the premier, our main renderfarm is totally broken… what now?…

 

  • Campbell & Koen want to copy some footage on a USB disk, but the server is configured to cache gigabytes, while Koen is illegally parked and wants to rush the files out to DELL but the disk won’t unmount, killing all processes that use the disk to force and unmount manages to kill _every_ process on the server including our render farm and studio internet connection, also in the panic some shelves got knocked over in our server room by accident with glassware breaking… We ended up copying most of the remaining files online.

 

  • Kjartan asks for a render hack to simplify some specific scene, the workaround only gets applied to one of our farms causing 5+ hour render times.

 

  • Francesco adds computers to the renderfarm without copying over the servers SSH key causing the renderfarm to lock up trying to login to a the system.

 

  • Our fallback system for storing footage during the shoot, mysteriously fails to boot the morning of the shoot,
    making things worse real life sysadmins on set are playing as `extras` in the first scene and have to get makeup and costumes done. – We manage without the backup system.

 

Things are eerily relaxed here at the studio a day before the premier, with luck we manage without too many problems this time.

– Campbell

  1. zomby says:

    mac and window? I thought that in Mango linux are used.

    • Sergey says:

      In the studio it’s indeed only Linux installed. But artists are working from their laptops quite often which is not necessarily carries Linux.

      • arrrharrr says:

        Can you give a recommendation which linux distro to use? Especially which one did you use with the GTX580 cards?

        I would really appreciate an answer to this :)

        • Sergey says:

          Hi.

          In the studio we’re using Ubuntu 12.04. It works pretty much nice and all hardware worked out of box i think. Can not remember any difficulties making it working for GTX580. Using Ubuntu is also nice if you would need stuff like latest Krita or so — pretty easy to find PPA with updates for software you want to try.

        • campbell says:

          recommend recent ubuntu WITHOUT a composting window manager, so gnome-fallback, xfce, openbox or so (all are used in the studio, but we dont use gnome-shell or unity because of issues using blender with composting window managers).

          But having access to ubuntu packages is handy.

          • arrrharrr says:

            Thanks Campbell and Sergej! That was exactly the info I was looking for. I managed to get a dedicated blender workstation/computer at work besides the mac pros (early 3.1) we are using and I wanted to have a linux system to be able to use the latest builds from the buildbot (since the mac builds there are months older). And yes the GTX580 did work fine after adding the x-swat ppa to 12.04. But blender was hardly usable with unity – for instance F-Keys bring up brightness and loudness applets and so on.

            However – It would be cool to have such information somewhere on the blender page or in the wiki (Might be I didn’t find it there).

          • SephirothTBM says:

            Hi Campbell thanks for sharing the info, currently i’m using ubuntu 12.04, with the gnome-shell and I haven’t had any trouble, could you share a lil’ more about the issues that you found, please!!! just curiosity!!! :P

          • campbell says:

            @arrrharrr, done

            http://wiki.blender.org/index.php/Doc:2.6/Manual/Introduction/Installing_Blender/Linux#Compositing_Desktop_Environments

            @SephirothTBM, some of the artists used unity and gnome shell, it worked mostly, but we did get strange drawing glitches with gnome-shell, cant recall if we had errors with unity.

            In another instance we found lots of windows would eat up texture memory, making cycles GPU render fail (since they share texture memory).

          • SephirothTBM says:

            Thanks for the info, I did some tests and well gnome-shell has been working very well alongside blender, about the memory it’s true I’m having some troubles while rendering using cycles, but I don’t have idea why it was caused, thanks for all you effort guys!!!!

  2. BernAr says:

    You guys rock!!! you’ll make it and we are proud of you :-)))

  3. J. says:

    No worries guys, I’d make waaaaaay more errors if you were to deploy me as a Linux sysadmin! :D

    What’s the fate of te Dell servers with the PDU on the fritz? :O

    See you tomorrow, guys!

    • Sergey says:

      Burned PDU was replaced, heavier fuses were installed all over, half of the nodes were connected to another electricity line.

      So far so good. It survived this weekend and keeps rendering. All 29 nodes of that farm are completely loaded with stuff to be rendered. Fingers continues to be crossed!

  4. Chris says:

    Haha, thanks for sharing the scary moments :) You’re right, these things happen to all teams, but you guys are the only ones that don’t have to pretend like they didn’t happen.

  5. Andreas says:

    i would love to see some of this stuff in the making of as well

  6. n-pigeon says:

    Hahaha, last photo killed me xD

  7. Robbie Losee says:

    Too funny. Love it.

  8. Psy-Fi says:

    Sergey in Gordon Freeman pose.

  9. fury_jin says:

    I hope this is 1 year problems list not 1 month…

    I guess that’s what you call “experience”

  10. shashank singh says:

    wow , just wow . Every sysadmin’s nightmare all thrown into one short sweet little package.

    +1 fury_jin

    This is invaluable experience :)

  11. Well it seems that you have fewer development fail moments. Comparing to Sintel or BBB you now have more system management moments. Good job guys!

  12. PhysicsGuy says:

    Had great fun at the pre-premiere yesterday. Amazing that with all the technical setbacks, you managed to put out such an epic movie! My wife, who is not a blenderhead or special effects fan, also greatly enjoyed the movie. So congratulations and thanks for all the hard work. It was awesome. Looking forward to the DVD!

  13. Ranjith Siji says:

    You guys rock!!! Congrats for mission critical works. we are proud of you :-)))

  14. Haha :D fun to read about… but some stuf made my stomach twist :)