| --- Log opened Thu Apr 29 00:00:22 2010 |
| 00:03 | | Derakon [Derakon@Nightstar-1ffd02e6.ucsf.edu] has quit [[NS] Quit: Leaving] |
| 00:10 | | shade_of_cpux is now known as cpux |
| 02:18 | | PinkFreud [WhyNot@NetworkAdministrator.Nightstar.Net] has joined #code |
| 02:26 | | Attilla [Attilla@FBC920.398CA6.FC4697.2D0785] has quit [Connection reset by peer] |
| 02:37 | | * Vornicus needs to actually, uh, interpolate, here, really. |
| 02:52 | | gnolam [lenin@Nightstar-38637aa0.priv.bahnhof.se] has quit [[NS] Quit: Z?] |
| 03:16 | | Thaqui [Thaqui@27B34E.D54D49.F53FA1.6A113C] has joined #code |
| 03:25 | <@McMartin> | INTERPOLATE |
| 03:26 | < Vornicus> | Victory! |
| 03:27 | < Vornicus> | Now if only I could choose marker and line formatting via data in Excel, this would be more useful! |
| 03:53 | < Vornicus> | http://www.flickr.com/photos/7861878@N06/4561660295/ |
| 04:38 | | Thaqui [Thaqui@27B34E.D54D49.F53FA1.6A113C] has quit [Client closed the connection] |
| 05:47 | | * Rhamphoryncus curses python set's overloading of > and < |
| 05:50 | | Serah [Z@26ECB6.A4B64C.298B52.D80DA0] has quit [Ping timeout: 121 seconds] |
| 05:52 | < Rhamphoryncus> | Writing a method that checks if I'm a superset. I want to return False under one condition early, and since I'm delegating most of the work to python's set I just take >= (the superset test) and reverse it, using <... |
| 06:00 | < Vornicus> | which is /strict/ subset. |
| 06:15 | | Serah [Z@3A600C.A966FF.5BF32D.8E7ABA] has joined #code |
| 06:32 | | celticminstrel [celticminstre@Nightstar-f8b608eb.cable.rogers.com] has quit [[NS] Quit: *hums* Can't stay now!] |
| 06:35 | | Vornicus is now known as Vornicus-Latens |
| 06:38 | < Rhamphoryncus> | where "struct subset" is also known as "completely fucking wrong" ;) |
| 06:41 | < Serah> | Vorn: Did I ever mention I love you? |
| 07:10 | | Thaqui [Thaqui@27B34E.D54D49.F53FA1.6A113C] has joined #code |
| 07:38 | | Serah [Z@3A600C.A966FF.5BF32D.8E7ABA] has quit [Ping timeout: 121 seconds] |
| 08:26 | <@McMartin> | http://inform7.com/news/2010/04/28/change-log-for-the-next-build/ |
| 08:46 | | You're now known as TheWatcher |
| 09:41 | | Attilla [Attilla@FBC920.398CA6.FC4697.2D0785] has joined #code |
| 09:41 | | mode/#code [+o Attilla] by Reiver |
| 10:02 | | Alek [omegaboot@Nightstar-7ff8f4eb.il.comcast.net] has quit [Ping timeout: 121 seconds] |
| 10:06 | | Alek [omegaboot@Nightstar-7ff8f4eb.il.comcast.net] has joined #code |
| 10:10 | | AnnoDomini [annodomini@Nightstar-7498e843.adsl.tpnet.pl] has joined #code |
| 10:10 | | mode/#code [+o AnnoDomini] by Reiver |
| 10:27 | | ToxicFrog [ToxicFrog@ServerAdministrator.Nightstar.Net] has quit [Connection reset by peer] |
| 10:32 | | Thaqui [Thaqui@27B34E.D54D49.F53FA1.6A113C] has quit [Connection closed] |
| 10:36 | | Rhamphoryncus [rhamph@Nightstar-8931f88f.abhsia.telus.net] has quit [Client exited] |
| 10:52 | | gnolam [lenin@Nightstar-38637aa0.priv.bahnhof.se] has joined #code |
| 10:55 | | Alek [omegaboot@Nightstar-7ff8f4eb.il.comcast.net] has quit [Client closed the connection] |
| 10:58 | | Alek [omegaboot@Nightstar-7ff8f4eb.il.comcast.net] has joined #code |
| 10:58 | | Syloqs-AFH [Syloq@NetworkAdministrator.Nightstar.Net] has quit [Ping timeout: 121 seconds] |
| 12:31 | | Orth [orthianz@Nightstar-bfd145ce.xnet.co.nz] has quit [Ping timeout: 121 seconds] |
| 12:33 | | GeekSoldier_ [Rob@Nightstar-e86e3e0d.ip.cablemo.net] has joined #code |
| 12:35 | | GeekSoldier [Rob@Nightstar-e86e3e0d.ip.cablemo.net] has quit [Ping timeout: 121 seconds] |
| 12:36 | | GeekSoldier_ [Rob@Nightstar-e86e3e0d.ip.cablemo.net] has quit [Connection reset by peer] |
| 12:36 | | GeekSoldier_ [Rob@Nightstar-e86e3e0d.ip.cablemo.net] has joined #code |
| 12:39 | | Orthia [orthianz@Nightstar-bfd145ce.xnet.co.nz] has joined #code |
| 13:46 | | Orthia [orthianz@Nightstar-bfd145ce.xnet.co.nz] has quit [Ping timeout: 121 seconds] |
| 13:47 | | Reiver [reaverta@ServerAdministrator.Nightstar.Net] has quit [Ping timeout: 121 seconds] |
| 13:54 | | Orthia [orthianz@Nightstar-bfd145ce.xnet.co.nz] has joined #code |
| 15:11 | | celticminstrel [celticminstre@Nightstar-f8b608eb.cable.rogers.com] has joined #code |
| 15:34 | | * gnolam stabs Linux UIs. |
| 15:43 | | ToxicFrog [ToxicFrog@ServerAdministrator.Nightstar.Net] has joined #code |
| 16:40 | | Serah [Z@26ECB6.A4B64C.298B52.D80DA0] has joined #code |
| 18:25 | | Derakon [Derakon@Nightstar-1ffd02e6.ucsf.edu] has joined #code |
| 18:26 | < Derakon> | It seems like there ought to be a better way to do this: |
| 18:26 | < Derakon> | dir = os.path.sep.join(filePath.split(os.path.sep)[:-1]) |
| 18:27 | < Derakon> | (The goal being to, given a path to a file, get the path of the directory the file is in) |
| 18:29 | < Derakon> | Ah ha! os.path.dirname() does the same thing. |
| 18:30 | < gnolam> | What? |
| 18:30 | < gnolam> | Hmm |
| 18:30 | | * gnolam 's backscroll had stuck. |
| 18:30 | < Derakon> | Heh. |
| 18:30 | < gnolam> | (So I was stuck on "[19:26] <Derakon> It seems like there ought to be a better way to do this:") |
| 18:31 | < Derakon> | (Idly, I do so love Python's array slice syntax. It's pretty.) |
| 18:31 | < celticminstrel> | Me too. |
| 18:32 | < celticminstrel> | Doing the same thing in Javascript is much more annoying. |
| 18:33 | < Derakon> | Well, assuming you don't want to do something like iterate backwards or get every other element, you can just use the slice() method on arrays in Javascript. |
| 18:39 | < celticminstrel> | Yes, but it's not as nice as Python. |
| 18:42 | < Derakon> | Well, no. |
| 18:47 | | ToxicFrog [ToxicFrog@ServerAdministrator.Nightstar.Net] has quit [Connection reset by peer] |
| 19:21 | | * Derakon ...s at the latest TDWTF. http://thedailywtf.com/Default.aspx |
| 19:28 | < gnolam> | "Poke a Dot"? |
| 19:29 | < Derakon> | Yeah, that one. |
| --- Log closed Thu Apr 29 19:31:19 2010 |
| --- Log opened Thu Apr 29 19:31:37 2010 |
| 19:31 | | TheWatcher [chris@Nightstar-b4529b0c.zen.co.uk] has joined #code |
| 19:31 | | Irssi: #code: Total of 9 nicks [0 ops, 0 halfops, 0 voices, 9 normal] |
| 19:31 | | Vornicus-Latens [vorn@ServerAdministrator.Nightstar.Net] has joined #code |
| 19:31 | | Attilla [Attilla@FBC920.398CA6.FC4697.2D0785] has joined #code |
| 19:32 | | Vornicus-Latens is now known as Vornicus |
| 19:32 | | Irssi: Join to #code was synced in 53 secs |
| 19:32 | | celticminstrel [celticminstre@Nightstar-f8b608eb.cable.rogers.com] has joined #code |
| 19:33 | | Kazriko [kaz@Nightstar-e09690fa.client.bresnan.net] has joined #code |
| 19:33 | | Rhamphoryncus [rhamph@Nightstar-8931f88f.abhsia.telus.net] has joined #code |
| 19:33 | | jerith [jerith@Nightstar-bf52129d.slipgate.za.net] has joined #code |
| 19:33 | | McMartin [mcmartin@Nightstar-3c130f1a.pltn13.sbcglobal.net] has joined #code |
| 19:33 | | Tarinaky [Tarinaky@Nightstar-83f5e5c6.adsl.virginmedia.net] has joined #code |
| 19:33 | | Taki^ [jwjw@Nightstar-9b459f81.consolidated.net] has joined #code |
| 19:33 | | Serah [Z@26ECB6.A4B64C.298B52.D80DA0] has joined #code |
| 19:33 | | AnnoDomini [annodomini@Nightstar-7498e843.adsl.tpnet.pl] has joined #code |
| 19:33 | | GeekSoldier_ [Rob@Nightstar-e86e3e0d.ip.cablemo.net] has joined #code |
| 19:33 | | ServerMode/#code [+ooo Kazriko McMartin AnnoDomini] by *.Nightstar.Net |
| 20:32 | | Syloqs_AFH [Syloq@NetworkAdministrator.Nightstar.Net] has joined #code |
| 20:33 | | Syloqs_AFH is now known as Syloqs-AFH |
| 20:34 | | Alek [omegaboot@Nightstar-7ff8f4eb.il.comcast.net] has quit [Ping timeout: 121 seconds] |
| 20:37 | | * gnolam stabs Linux again. |
| 20:53 | | ToxicFrog [ToxicFrog@ServerAdministrator.Nightstar.Net] has joined #code |
| 21:02 | | AnnoDomini [annodomini@Nightstar-7498e843.adsl.tpnet.pl] has quit [Ping timeout: 121 seconds] |
| 21:04 | | AnnoDomini [annodomini@Nightstar-828f7ad0.adsl.tpnet.pl] has joined #code |
| 21:26 | | Alek [omegaboot@Nightstar-c5f3565b.il.comcast.net] has joined #code |
| 22:01 | < Derakon> | Got a design question for you folks. |
| 22:02 | < Derakon> | The microscope has generated several terabytes of data so far. |
| 22:02 | < Derakon> | Most of that is stored on a selection of USB external hard drives. |
| 22:02 | < Derakon> | Shortly after I arrived, I discovered that the data being generated is very amenable to compression, and I was thinking it'd be a good idea to set up a cataloguing system. |
| 22:03 | < Derakon> | The question becomes, how to maintain the catalogue? |
| 22:03 | < Derakon> | I don't think we can hope to process all the data we've already generated, for example. |
| 22:03 | < gnolam> | 00, |
| 22:03 | < gnolam> | - |
| 22:03 | < gnolam> | Err |
| 22:03 | < gnolam> | Stuff falling on my keyboard there. |
| 22:03 | < Derakon> | Heh. |
| 22:04 | < gnolam> | This is probably a sign that I should unclutter my desk a bit... |
| 22:04 | < Derakon> | So what I'm looking for is a system that accepts data, compresses it, generates a thumbnail and viewable metadata, lets you store the data elsewhere, and then tells you where it is. |
| 22:04 | < Derakon> | But there's so many options fon a desynch there if you move the data without informing the catalogue... |
| 22:06 | < Derakon> | For that matter, telling the catalogue you're going to put data somewhere, and actually putting it there, are entirely different things. |
| 22:07 | < Derakon> | Maybe a system where you plug in external drives to the catalogue computer, it fills them for you, and it remembers what it put there? But the biologists do their processing on separate computers... |
| 22:08 | < Derakon> | I suppose we could rely on the catalogue just for backups. |
| 22:09 | < Derakon> | With the implication being that you would always copy from the catalogued drives, never write to them except through the catalogue system. |
| 22:13 | < Derakon> | Thoughts on this design? |
| 22:39 | | AnnoDomini [annodomini@Nightstar-828f7ad0.adsl.tpnet.pl] has quit [[NS] Quit: Gnar.] |
| 22:46 | < Rhamphoryncus> | What's the purpose of the catalogue? |
| 22:47 | < Derakon> | To make it easier to find data that's been shelved on an external HD for more than a couple of weeks. |
| 22:47 | < Rhamphoryncus> | Since things are distributed and mobile, why not give each drive its own catalogue, then have a metacatalogue that tracks them? |
| 22:48 | < Rhamphoryncus> | Updating the metacatalogue can be done after something is moved |
| 22:48 | < Derakon> | The goal is to have a single point you can go to and say "Where is File X?" and it will tell you which HD it's on (and where on the HD). |
| 22:48 | < Rhamphoryncus> | The metacatalogue can also include information like drive IDs as well as the last seen location of that drive |
| 22:48 | < Derakon> | I'm not clear how having catalogues on each HD helps with that. |
| 22:48 | < Rhamphoryncus> | Sync time :) |
| 22:49 | < Derakon> | And I'm assuming that the biologists will be able to keep track of where the drives are. |
| 22:49 | < Rhamphoryncus> | The thumbnail should already be generated on each drive. You can also store a hash of the data |
| 22:52 | < Rhamphoryncus> | It's more of a client-server model, where each client is storing the data and telling the server what it has |
| 22:52 | < Rhamphoryncus> | (as opposed to being a pure slave to the server) |
| 22:54 | < Rhamphoryncus> | hrm. It doesn't entirely solve the desync problem, but it does mean someone could move a file, then run a program that updates the local catalogue, followed by pushing the update to the central catalogue |
| 23:10 | < Derakon> | Heh. What we really need is a smart version control system, I guess. |
| 23:10 | < Rhamphoryncus> | pretty much |
| 23:12 | < Rhamphoryncus> | Or alternatively, scrap the usb drives idea and turn the whole thing in a central networked filesystem |
| 23:27 | < Derakon> | That would be a ton of work. |
| 23:27 | < Derakon> | And centralizing file access isn't helpful because the biologists need to be able to do their analysis remotely. |
| 23:39 | < Derakon> | Argh I am surrounded by coughing people. >.< |
| 23:46 | < Rhamphoryncus> | They can have a remote cache, but the canonical copy should be central |
| 23:47 | < Rhamphoryncus> | I concede though, it's a heck of a lot of work to change now |
| 23:47 | < Derakon> | Yeah, I'd guess we have at least 10TB of data in this room alone. |
| 23:52 | < Rhamphoryncus> | .. that's 5 drives |
| 23:53 | < Derakon> | Well, more then. I don't know how big the drives are. |
| 23:54 | < Rhamphoryncus> | The drives are probably older and smaller |
| 23:54 | < Derakon> | But I can see 8 external HDs, and two tower computers. |
| 23:56 | < Rhamphoryncus> | heh, does this make you feel small? http://news.bbc.co.uk/2/hi/technology/6425975.stm |
| 23:57 | < Rhamphoryncus> | mailing(!) 120 terabyte blocks to and from google |
| 23:57 | < Derakon> | Never underestimate the bandwidth of a 747 filled with phone books. |
| 23:57 | < Rhamphoryncus> | heh |
| 23:58 | < Derakon> | 120TB in, oh, two weeks is ~99MBps. |
| 23:59 | < Rhamphoryncus> | This whole problem has a smell to it. It's been solved a thousand times over. You shouldn't have to solve it again, you should be able to get a contract where someone provides the drives and RAID for onsite backup, as well as offsite |
| 23:59 | < Derakon> | (or 106MiBps) |
| 23:59 | < Derakon> | Rhamph: well, the details of maintaining metadata are going to be pretty unique to our domain space. |
| 23:59 | < Rhamphoryncus> | Sure, but that's metadata. That's a layer up |
| --- Log closed Fri Apr 30 00:00:06 2010 |