| --- Log opened Fri Sep 04 00:00:24 2009 |
| --- Day changed Fri Sep 04 2009 |
| 00:00 | < Rhamphoryncus> | or then again, delete the if statements entirely |
| 00:07 | < Rhamphoryncus> | if it was an exception from python code I'd treat it as an abnormal exit and want it to be an if statement at the top, even if there was no normal exit path |
| 00:07 | < Rhamphoryncus> | But if it's just a variant of "work's done, let's exit" then the while loop is better |
| 00:08 | < Derakon[work]> | Well, if I can swing it I'll switch over to a saner thread implementation. |
| 00:10 | | Attilla [~The.Attil@92.21.240.ns-3266] has quit [Ping Timeout] |
| 00:15 | | Attilla [~The.Attil@92.9.58.ns-27495] has joined #code |
| 00:15 | | mode/#code [+o Attilla] by ChanServ |
| 00:26 | | * Derakon[work] finishes reading the last .cpp file in the project. |
| 00:26 | < Derakon[work]> | And I'm now confident that the only thing keeping me from being able to rewrite this in pure Python is the threading. |
| 00:31 | | You're now known as TheWatcher[T-2] |
| 00:37 | | You're now known as TheWatcher[zZzZ] |
| 00:49 | | Derakon[work] [~Derakon@Nightstar-6042.ucsf.edu] has quit [Quit: Leaving] |
| 01:53 | | Derakon[AFK] is now known as Derakon |
| 02:45 | < Rhamphoryncus> | Derakon: how much concurrency does it need? |
| 02:46 | <@Derakon> | I'm not certain, and that will bear investigation. |
| 02:46 | <@Derakon> | The main thing is the ability to write images to disk while receiving other images over a socket. |
| 02:46 | <@Derakon> | To my knowledge, nothing else in the software is remotely time-sensitive, but images come in at a quoted rate of about 6MB/s. |
| 02:52 | <@Vornicus> | Htat's still not a very small feat, 6MB/s. You may need to keep that section in C/C++. |
| 02:52 | <@Vornicus> | ...on the other hand the reference implementation of BT is in Python. |
| 02:52 | <@Derakon> | Yeah, I know some segments may remain in C/C++. |
| 02:53 | <@Derakon> | The important thing, IMO, is to turn this from a C++ program that has an absolutely massive amount of connected Python, to a Python program that has a small amount of connected C++. |
| 02:55 | < Rhamphoryncus> | the reference implementation of BT is vile |
| 02:56 | < Rhamphoryncus> | Other than being able to cancel blocked I/O you might actually do well with python's threading |
| 02:57 | < Rhamphoryncus> | And heck, if the disk isn't a bottleneck you might do well with a single-threaded poll-based server |
| 02:57 | <@Derakon> | Heh. |
| 02:57 | <@Vornicus> | Rham: Vile perhaps, but still Python. |
| 02:57 | <@Vornicus> | And I almost called you the white house chief of staff. |
| 02:58 | <@Derakon> | ...? |
| 02:58 | < Rhamphoryncus> | How much scalability? One connection at 6 MB/s, or possibly many each at 6 MB/s? And does each do their own processing and disk writing, or is there a lot of intermingling? |
| 02:58 | <@Derakon> | Oh, Rahm. |
| 02:58 | | * Rhamphoryncus hopes he isn't being too chatty ;) |
| 02:59 | <@Derakon> | Rhamphoryncus: Er, I'm not entirely certain...I think the quoted rate was for all four cameras going full-bore, but I could be wrong. |
| 02:59 | <@Derakon> | There's one thread for each camera, and one thread for doing disk I/O, though. |
| 02:59 | < Rhamphoryncus> | They get combined into a single file? |
| 02:59 | <@Derakon> | The camera threads feed into the ring buffer; the disk thread consumes the buffer. |
| 02:59 | <@Derakon> | Yeah. |
| 03:00 | < Rhamphoryncus> | ahh okay |
| 03:00 | | * Rhamphoryncus wonders if linux still bullshits writability of the disk |
| 03:01 | < Rhamphoryncus> | Something about it always being a small wait, compared to the network, so why bother telling the app it's not ready yet.. |
| 03:02 | < Rhamphoryncus> | Your app sounds like just enough to make me worry about python's concurrency support |
| 03:03 | <@Derakon> | What exactly is bad about Python threads? |
| 03:03 | <@Derakon> | Large context-switch costs? |
| 03:03 | <@Derakon> | Slow mutexes? |
| 03:05 | < Rhamphoryncus> | GIL. single lock of the entire interpreter |
| 03:05 | < Rhamphoryncus> | lots of crappy interface stuff too, but the GIL kills performance |
| 03:06 | <@Derakon> | Ahh, which basically means Python is currently doomed to ignore auxiliary cores. |
| 03:06 | <@Derakon> | So threads mainly buy you simplicity in handling context. |
| 03:06 | | Attilla [~The.Attil@92.9.58.ns-27495] has quit [Quit: ] |
| 03:08 | <@Vornicus> | At the cost of (for instance) race conditions. |
| 03:08 | < Rhamphoryncus> | yup. And in fact it can perform worse on two cores than on one |
| 03:08 | < Rhamphoryncus> | However, if you spend enough of your time in C/C++ it can work fine anyway |
| 03:12 | <@Derakon> | I wonder if the threads could be entirely C/C++ threads, with the Python side occasionally polling them to see if they're done... |
| 03:12 | < Rhamphoryncus> | (Amdahl's law applies) |
| 03:13 | < Rhamphoryncus> | That'd be nice if possible |
| 03:15 | <@Derakon> | Well, maybe on Monday I'll find out if the extra threads are actually required. |
| 03:16 | <@Derakon> | I wouldn't put it past the original author to go for multithreading without checking if it's necessary first. |
| 03:17 | < Rhamphoryncus> | Well... from a theoretical PoV I think threads are the right solution. This is concurrency (even if you do event-driven), and threads are the ideal solution. It's just that so many languages *SUCK* at it |
| 03:18 | <@Derakon> | Including, sadly, Python. |
| 03:19 | < Rhamphoryncus> | yup |
| 03:19 | < Rhamphoryncus> | Have I done my spiel about python-safethread for you? |
| 03:19 | <@Derakon> | No, I don't think you have. |
| 03:20 | < Rhamphoryncus> | I forked CPython, redesigned the threading to be both safe and easy, allowed it in the design to scale up, actually removed the GIL.. but my attempt at boding the refcounting to scale didn't work. I'd need to go and rewrite most of it to truly use a tracing GC |
| 03:21 | <@Derakon> | Boding the refcounting? |
| 03:23 | < Rhamphoryncus> | bodging* |
| 03:23 | < Rhamphoryncus> | I came up with some fancy coalescing buffering scheme |
| 03:24 | <@Derakon> | I guess cross-thread refcounting gets tricky fast, huh? |
| 03:24 | < Rhamphoryncus> | I actually got around 120% throughput total on a dual core laptop. (Meaning two threads were slightly faster than unmodified CPython single thread. On a trivial benchmark) |
| 03:24 | < Rhamphoryncus> | They contend over the cache lines. Simple as that |
| 03:26 | < Rhamphoryncus> | Thing is, it's not a conceptually hard problem to fix. I did all the conceptually hard problems. All I'd need is a different base, like Jython. Or a team to rewrite from scratch |
| 03:27 | <@Derakon> | So CPython is just fundamentally unable to handle it with its current GC? |
| 03:27 | < Rhamphoryncus> | Basically |
| 03:28 | < Rhamphoryncus> | I do have some hope that unladen swallow will make ground.. if they're doing JIT of pure python code then those parts could use a different GC, while all the libraries use a unscalable fallback API |
| 03:34 | < Rhamphoryncus> | My current project is a static analyzer of python code, ie the groundwork for a true python compiler |
| 03:38 | <@Derakon> | Do you do this for your day job, or is it a hobby? |
| 03:41 | < Rhamphoryncus> | hobby |
| 03:41 | <@Vornicus> | Rham is a crazy fucker, but I think we knew this. :) |
| 03:41 | < Rhamphoryncus> | I do have a reputation to uphold *g* |
| 03:42 | <@Derakon> | Well, your efforts are appreciated. :) |
| 03:42 | < Rhamphoryncus> | :) |
| 03:43 | <@Vornicus> | Reminds me, I still wanted to make a sorted list thingy. |
| 03:44 | < Rhamphoryncus> | with a uniqueness constraint? |
| 03:44 | <@Vornicus> | Or, finish the one I have; I need to add support for reverse, and make it take up less space when you don't pass it a key function, and then eventually see about Cifying it. |
| 03:44 | <@Vornicus> | Rham: no. |
| 03:44 | < Rhamphoryncus> | wait, sorted, not ordered >.> |
| 03:45 | < Rhamphoryncus> | What's it do? Maintain the sort invariant as you add items? |
| 03:46 | <@Vornicus> | Indeed. |
| 03:46 | < Rhamphoryncus> | what's the big-O? |
| 03:47 | <@Vornicus> | Big-O on insertion and deletion is the same as existing list insertion and deletion: O(number of elements that would need to be moved) |
| 03:47 | <@Vornicus> | Unless of course the key func dominates; we're talking logn comparisons. |
| 03:47 | < Rhamphoryncus> | oh, O(n) |
| 03:47 | < Rhamphoryncus> | looked at heapq? |
| 03:48 | <@Vornicus> | Yeah. Completely unacceptable; I need to be able to index into it live. |
| 03:48 | < Rhamphoryncus> | ah |
| 03:48 | <@Vornicus> | In addition to its lack of key support. |
| 03:48 | < Rhamphoryncus> | I think a wrapper would give you that |
| 03:49 | <@Vornicus> | Using sort() repeatedly also sucks: it has to regenerate the keys each time. |
| 03:49 | < Rhamphoryncus> | oh, you want it to cache the keys.. that can also be done with a wrapper |
| 03:50 | <@Vornicus> | It is mostly a wrapper. |
| 03:50 | < Rhamphoryncus> | I mean on each object |
| 03:50 | <@Vornicus> | ah |
| 03:51 | <@Vornicus> | Stability here is another trick: From what I can tell, a stable sort would require the wrapper to also get an insertion id number as the second element of the tuple. |
| 03:52 | < Rhamphoryncus> | python's builtin sort has been stable for a few versions now |
| 03:52 | <@Vornicus> | Right. |
| 03:52 | < Rhamphoryncus> | Not that you'd be using it. You'd be using bisect |
| 03:53 | <@Vornicus> | But maintaining that stability is the problem: python's builtin sort works off the key only, if the key exists. |
| 03:53 | < Rhamphoryncus> | hmm? Is your key changing? |
| 03:54 | <@Vornicus> | Wrapping it, as far as I can tell, requires that I either write a custom comparator (slow) or using a tuple (would put the content in the sort path unless I have the [1] element being an insertion id) |
| 03:55 | < Rhamphoryncus> | You could store a tuple, then have a trivial key function to get only the first item from it |
| 03:56 | < Rhamphoryncus> | said function even exists in C. operator.itemgetter(0) |
| 03:56 | <@Vornicus> | Hm.... yes, that would do it. |
| 03:57 | < Rhamphoryncus> | And anyway, if insertion performance was really an issue you'd use a tree instead. Get O(log n) rather than O(n) |
| 03:58 | <@Vornicus> | Well, I mostly want insertion performance to not have to regenerate the keys each time (because this is Frowned Upon; my bisect upgrade was rejected because of it) |
| 03:59 | | * Rhamphoryncus ndos |
| 03:59 | < Rhamphoryncus> | hrm |
| 03:59 | <@Vornicus> | Indeed, that I kind of have to use bisect in the first place invalidates your "trivial key function" thing - I can't keyify bisect and get it accepted. |
| 04:00 | < Rhamphoryncus> | So you can use list.sort and a trivial key function, no comparator, or bisect, with a comparator |
| 04:00 | <@Vornicus> | bisect doesn't have a comparator, unless you mean writing a custom one. |
| 04:00 | < Rhamphoryncus> | But for a large enough list you'll be better with bisect anyway |
| 04:00 | < Rhamphoryncus> | yes |
| 04:01 | <@Vornicus> | For the object wrapper class. |
| 04:01 | < Rhamphoryncus> | yup |
| 04:01 | <@Vornicus> | hrmng. :/ |
| 04:01 | < Rhamphoryncus> | want me to write it for you? ;) |
| 04:01 | <@Vornicus> | I see how it would work, I think. I just plain don't like it. |
| 04:01 | <@Vornicus> | And isn't __cmp__ deprecated for the rich comparators? |
| 04:01 | < Rhamphoryncus> | yes |
| 04:02 | <@Vornicus> | THough now that I think on it there's the mixin thingy. |
| 04:02 | < Rhamphoryncus> | eh overkill, if you know which one bisect wants |
| 04:03 | <@Vornicus> | Though that still leaves the question of reverse support. |
| 04:04 | < Rhamphoryncus> | http://paste.ubuntu.com/264755/ |
| 04:04 | < Rhamphoryncus> | reversed() builtin? |
| 04:05 | < Rhamphoryncus> | Depending on what you need to do with it while reversed |
| 04:05 | <@Derakon> | Hm, __repr__ is serialize? |
| 04:05 | <@Vornicus> | __repr__ is serialize; __str__ is prettyprint. |
| 04:05 | | * Derakon nods. |
| 04:06 | <@Derakon> | More to the point, __str__ has no obligation to be valid Python. |
| 04:06 | < Rhamphoryncus> | heh, bisect docs suggest using a parallel list for the key |
| 04:07 | < Rhamphoryncus> | __repr__ has no obligation either. It's just a convention to be reversible |
| 04:07 | <@Vornicus> | The trick here, Rham, is that I want it to act - as far as is possible - like I appended the item to the list and ran sort() on it with the same parameters as originally. |
| 04:07 | <@Vornicus> | You still can't reverse a __repr__'d file object. |
| 04:08 | <@Vornicus> | Rhamphoryncus: actually that's the trick I'm using now. |
| 04:08 | < Rhamphoryncus> | Vornicus: So if you sorted it in reverse in the first place you want that? |
| 04:08 | <@Vornicus> | Rham: exactly. |
| 04:08 | < Rhamphoryncus> | sounds a little silly ;) |
| 04:09 | <@Vornicus> | I'm being picky here because I sat down and figured out my use case and this is it. |
| 04:12 | | * Rhamphoryncus nods |
| 04:25 | <@Vornicus> | And at that I figure if I can pull this off, people will find my care useful. :) |
| 04:29 | | gnolam [lenin@Nightstar-1382.A163.priv.bahnhof.se] has joined #Code |
| 04:29 | | mode/#code [+o gnolam] by ChanServ |
| 04:29 | | gnolam is now known as gninsomniac |
| 04:38 | | Vornicus [~vorn@ServicesOp.Nightstar.Net] has quit [Quit: Leaving] |
| 04:39 | | Orthia [~Orthianz@Nightstar-9186.xdsl.xnet.co.nz] has left #Code [] |
| 04:41 | | Vornicus [Vornicus@Admin.Nightstar.Net] has joined #code |
| 04:41 | | mode/#code [+o Vornicus] by ChanServ |
| 04:43 | | * Derakon eyes ping. "2, 3, 4, 5, ..., 17, 18, 19, 20..." |
| 04:44 | | * Derakon flails at Tcher with http://www.maliki.com/strip.php?strip=164 |
| 04:44 | <@Derakon> | Mischan. |
| 05:16 | | SmithKurosaki [~Smith@Nightstar-10871.dsl.teksavvy.com] has quit [Operation timed out] |
| 05:24 | | Derakon [~Derakon@Nightstar-4912.hsd1.ca.comcast.net] has quit [Ping Timeout] |
| 05:27 | | Derakon [~Derakon@Nightstar-4912.hsd1.ca.comcast.net] has joined #code |
| 05:27 | | mode/#code [+o Derakon] by ChanServ |
| 05:32 | | Thaqui [~Thaqui@121.98.166.ns-22683] has joined #code |
| 05:32 | | mode/#code [+o Thaqui] by ChanServ |
| 05:34 | | Derakon [~Derakon@Nightstar-4912.hsd1.ca.comcast.net] has quit [Operation timed out] |
| 05:45 | | SmithKurosaki [~Smith@Nightstar-5507.dsl.teksavvy.com] has joined #code |
| 05:45 | | mode/#code [+o SmithKurosaki] by ChanServ |
| 06:10 | | Consul [~Consul__@Nightstar-1744.dsl.sfldmi.ameritech.net] has quit [Quit: Leaving] |
| 09:00 | | You're now known as TheWatcher |
| 10:07 | | Rhamphoryncus [~rhamph@Nightstar-3964.abhsia.telus.net] has quit [Quit: Rhamphoryncus] |
| 10:39 | | SmithKurosaki [~Smith@Nightstar-5507.dsl.teksavvy.com] has quit [Operation timed out] |
| 10:47 | | MyCatVerbs [~mycatverb@Nightstar-13709.lurkingfox.co.uk] has quit [Ping Timeout] |
| 10:51 | | SmithKurosaki [~Smith@Nightstar-7134.dsl.teksavvy.com] has joined #code |
| 10:51 | | mode/#code [+o SmithKurosaki] by ChanServ |
| 10:54 | | SmithKurosaki [~Smith@Nightstar-7134.dsl.teksavvy.com] has quit [Operation timed out] |
| 10:59 | | MyCatVerbs [~mycatverb@Nightstar-13709.lurkingfox.co.uk] has joined #code |
| 11:00 | | mode/#code [+o MyCatVerbs] by ChanServ |
| 11:00 | | SmithKurosaki [~Smith@Nightstar-10289.dsl.teksavvy.com] has joined #code |
| 11:00 | | mode/#code [+o SmithKurosaki] by ChanServ |
| 12:04 | | AnnoDomini [AnnoDomini@Nightstar-28353.neoplus.adsl.tpnet.pl] has quit [Quit: Some people find sanity a little confining.] |
| 12:11 | | AnnoDomini [AnnoDomini@Nightstar-28353.neoplus.adsl.tpnet.pl] has joined #Code |
| 12:11 | | mode/#code [+o AnnoDomini] by ChanServ |
| 12:22 | | Attilla [~The.Attil@92.9.58.ns-27495] has joined #code |
| 12:22 | | mode/#code [+o Attilla] by ChanServ |
| 13:49 | | Consul [~Consul__@Nightstar-1744.dsl.sfldmi.ameritech.net] has joined #code |
| 13:49 | | mode/#code [+o Consul] by ChanServ |
| 14:27 | | Netsplit DeepThought.NY.US.Nightstar.Net <-> Troika.TX.US.Nightstar.Net quits: @MyCatVerbs, @Vornicus, @Attilla, @AnnoDomini, @SmithKurosaki |
| 14:27 | | Netsplit DeepThought.NY.US.Nightstar.Net <-> Troika.TX.US.Nightstar.Net quits: @gninsomniac, @Reiver, crem, Kazriko, @GeekSoldier, Chi, @jerith, @Chalain |
| 16:34 | | Chalain [~chalain@Admin.Nightstar.Net] has joined #code |
| 16:34 | | mode/#code [+o Chalain] by ChanServ |
| 16:34 | | jerith [~jerith@IRCop.Nightstar.Net] has joined #code |
| 16:34 | | mode/#code [+o jerith] by ChanServ |
| 16:34 | | Chi [~omegaboot@Nightstar-5572.dsl.emhril.sbcglobal.net] has joined #code |
| 16:34 | | Reiver [~reaverta@Admin.Nightstar.Net] has joined #code |
| 16:34 | | Kazriko [~kaz@Nightstar-26123.gdj-co.client.bresnan.net] has joined #code |
| 16:34 | | crem [~moo@Nightstar-28703.adsl.mgts.by] has joined #code |
| 16:34 | | GeekSoldier [~Rob@Nightstar-8573.midstate.ip.cablemo.net] has joined #code |
| 16:34 | | mode/#code [+o Reiver] by ChanServ |
| 16:34 | | gninsomniac [lenin@Nightstar-1382.A163.priv.bahnhof.se] has joined #code |
| 16:34 | | Vornicus [Vornicus@Admin.Nightstar.Net] has joined #code |
| 16:34 | | mode/#code [+o GeekSoldier] by ChanServ |
| 16:34 | | MyCatVerbs [~mycatverb@Nightstar-13709.lurkingfox.co.uk] has joined #code |
| 16:34 | | SmithKurosaki [~Smith@Nightstar-10289.dsl.teksavvy.com] has joined #code |
| 16:34 | | mode/#code [+o Vornicus] by ChanServ |
| 16:34 | | mode/#code [+o MyCatVerbs] by ChanServ |
| 16:34 | | Attilla [~The.Attil@92.9.58.ns-27495] has joined #code |
| 16:34 | | mode/#code [+o SmithKurosaki] by ChanServ |
| 16:34 | | AnnoDomini [AnnoDomini@Nightstar-29066.neoplus.adsl.tpnet.pl] has joined #code |
| 16:34 | | Derakon[work] [~Derakon@Nightstar-6042.ucsf.edu] has joined #code |
| 16:34 | | ServerMode/#code [+o gninsomniac] by Troika.TX.US.Nightstar.Net |
| 16:34 | | mode/#code [+o Attilla] by ChanServ |
| 16:34 | | mode/#code [-o gninsomniac] by ChanServ |
| 16:39 | < Chi> | bah. |
| 16:39 | < Chi> | tell you what, I'll do it today. |
| 16:40 | < AnnoDomini> | Tell me when you're ready, then I'll exhume the thread from the Neo-Popedome. |
| 17:13 | < Chi> | I'll start in an hour or so. |
| 17:13 | < Chi> | I'll need the original thread, on the original board, for research. -_- |
| 17:14 | < Chi> | and I'll need the sidequest thread to make running post in. |
| 17:15 | < AnnoDomini> | The old forum is still up. I've been seeing to that. |
| 17:15 | < Chi> | thanks. |
| 17:15 | < Chi> | ok, I gotta go for now. |
| 17:37 | | * gninsomniac feels he's missing something in that conversation. |
| 17:39 | | Thaqui [~Thaqui@121.98.166.ns-22683] has quit [Client exited] |
| 17:39 | < AnnoDomini> | It's an off-topic, inside convo. :p |
| 17:55 | | Reiver [~reaverta@Admin.Nightstar.Net] has quit [Ping Timeout] |
| 18:00 | < AnnoDomini> | Thread moved. |
| 18:01 | | NSGuest-2201 [~reaverta@Admin.Nightstar.Net] has joined #Code |
| 18:05 | < Derakon[work]> | Bahhh...the resource module doesn't seem to be reporting memory usage properly. |
| 18:05 | < Derakon[work]> | Any suggestions for an OS-independent way to get the current process size? |
| 18:20 | < gninsomniac> | There isn't really one. |
| 18:32 | < Derakon[work]> | Yay. |
| 18:33 | < Derakon[work]> | I found the resource module, but testing it on my lappy I got zeroes for memory size even while top reported over 1GB allocated to the process... |
| 18:37 | < gninsomniac> | Trying to find out the exact amount of memory used by a program is pretty much a lost cause nowadays anyway. |
| 18:37 | < Derakon[work]> | I don't need exact amounts; I'm looking for trends over time. |
| 18:37 | < Derakon[work]> | That said, PHP has a function to get the amount of memory it's using which tracked pretty well to what top reported, when I had cause to use it. |
| 18:39 | | Finale [c0cb88fe@Nightstar-14595.mibbit.com] has joined #code |
| 18:47 | | Vornicus [Vornicus@Admin.Nightstar.Net] has quit [Quit: ] |
| 18:47 | < Derakon[work]> | Anyway, what I'm dealing with here is an irregular crash issue. |
| 18:48 | < Derakon[work]> | The software running the micrsocope will occasionally, during long-running experiments, crash. Sometimes it happens inside half an hour, sometimes it'll make it five hours. One of the researchers said he managed to fairly consistently get it to crash within 15-20 minutes or so. |
| 18:48 | < Derakon[work]> | An informal survey using Windows process manager indicates that memory usage is not trending upwards; the machine has 4GB of RAM and the process is currently using about 124MB. |
| 18:49 | < Derakon[work]> | So it doesn't look like a memory leak. |
| 18:50 | < Derakon[work]> | Currently I'm hypothesizing memory corruption, which won't be any fun at all to track down. |
| 18:50 | | * Derakon[work] idly wonders if it's possible for the Python half of a C++/Python app to clobber memory. |
| 18:51 | < Finale> | how about the Monty Python half? |
| 18:51 | | * Derakon[work] has Finale taken out and shot. |
| 18:53 | < Derakon[work]> | Based on the log messages, the error occurs on one of these two lines: |
| 18:53 | < Derakon[work]> | X.f_mrc_hdr.NumTimes = nTimes |
| 18:53 | < Derakon[work]> | X.f_mrc_hdr.Num = omx.getNX(), omx.getNY(), nZ*nTimes*nw |
| 18:53 | < Derakon[work]> | Which look innocuous. |
| 18:53 | < Finale> | Help! Help! I'm being repressed! |
| 18:58 | < AnnoDomini> | Finale: Thread's up. |
| 18:58 | | Rhamphoryncus [~rhamph@Nightstar-3964.abhsia.telus.net] has joined #code |
| 18:59 | < Derakon[work]> | Hey Rhamphoryncus, do you know if, in a combined C++/Python program, the Python side can cause memory corruption? |
| 19:00 | < Finale> | thanks, AD. reading now. I START in the original thread, and have to continue to 1.5 in the new board. -_- |
| 19:00 | < Derakon[work]> | I'm trying to track down an irregular crash. |
| 19:01 | < AnnoDomini> | Finale: You can probably use the TL;DRs at the start of the sequels. |
| 19:04 | < Finale> | ehh. |
| 19:05 | < Finale> | I need the practice reading and condensing myself. -_- |
| 19:05 | < Finale> | oh my bad. no 1.5 |
| 19:05 | < Finale> | #2 is still on the old board, THEN we have 2.5 on the new one. |
| 19:10 | | * Derakon[work] shakes his fist at http://paste.ubuntu.com/265097/ |
| 19:13 | < Finale> | AD, when's Erik due to come on, just out of curiousity? >_> |
| 19:14 | < AnnoDomini> | He pinged out some time ago. |
| 19:15 | < AnnoDomini> | If he doesn't get back in like 2 hours, the soonest we'll see of him is Monday, since he's going for a church picnic or something. |
| 19:15 | < Finale> | ahh. |
| 19:15 | < Finale> | one long picnic, that. >_> |
| 19:15 | < Derakon[work]> | Could you guys do this over msg or something, please? |
| 19:16 | < Derakon[work]> | It really has nothing whatsoever to do with #code. |
| 19:16 | < Derakon[work]> | And I don't think any of the rest of us are interested. |
| 19:16 | < Finale> | let's go back to #dnd, AD. :P |
| 19:18 | < AnnoDomini> | Fine. |
| 21:21 | < Rhamphoryncus> | Derakon[work]: sorry, roach in keyboard |
| 21:21 | < Rhamphoryncus> | No, python code should never cause memory corruption |
| 21:22 | < Rhamphoryncus> | It's usually a library that screws something up (like refcounting), then it shows up later in python code |
| 21:23 | < Derakon[work]> | That's what I was afraid of. |
| 21:23 | < Derakon[work]> | (Roach in keyboard? Literally?) |
| 21:23 | < Derakon[work]> | Assuming this is a memory corruption bug, the problem could be anywhere. |
| 21:27 | < Rhamphoryncus> | literally. Spent 2 hours disassembling, cleaning, and reassembling. Didn't find the bugger |
| 21:28 | < Derakon[work]> | Yich. |
| 21:28 | < Rhamphoryncus> | Fairly common is refcounting bugs. Sometimes you can get rid of a DECREF, which'll cause a leak, but help you narrow down the cause |
| 21:28 | < Derakon[work]> | Hmm, thanks, will look into that. |
| 21:29 | < Derakon[work]> | There are four calls to DECREF in the code right now... |
| 21:29 | < Rhamphoryncus> | .. heh |
| 21:29 | < Rhamphoryncus> | any that'd be better using Py_CLEAR? |
| 21:30 | < Rhamphoryncus> | (decref and set to NULL) |
| 21:30 | < Derakon[work]> | No idea... |
| 21:30 | < Derakon[work]> | I've never worked with Python from this direction before. |
| 21:30 | < Derakon[work]> | And I only barely know what's going on in this code in the first place. |
| 21:30 | | * Rhamphoryncus nods |
| 21:30 | < Derakon[work]> | (Which is to say, I think I know what it's doing overall. I could be wrong!) |
| 21:31 | < Rhamphoryncus> | incidentally, I did end up with a spare screw ;) |
| 21:31 | < Derakon[work]> | One of the DECREF calls is on the result of calling PyRun_String("execfile(\'somefile\')") |
| 21:32 | < Rhamphoryncus> | heh |
| 21:32 | < Derakon[work]> | Another is the result of calling PyImport_ImportModule("__main__") |
| 21:33 | < Derakon[work]> | ...those are both XDECREF, actually. |
| 21:33 | < Rhamphoryncus> | that just has a check for NULL first |
| 21:33 | < Derakon[work]> | Ah. |
| 21:34 | < Derakon[work]> | And the other two are in a function named callPyOnExit(), so I wouldn't think they'd be a big issue... |
| 21:34 | | Finale [c0cb88fe@Nightstar-14595.mibbit.com] has left #code [] |
| 21:34 | < Derakon[work]> | ...well, but for that matter, the first two are in a function named OnInit(). |
| 21:35 | < Rhamphoryncus> | Are you calling python from multiple threads? |
| 21:35 | < Derakon[work]> | ...I don't think so? |
| 21:35 | < Derakon[work]> | There's also 105 calls to DECREF in some generated SWIG code. |
| 21:39 | < Rhamphoryncus> | could add a print from whatever's your toplevel python code. pthread_self() returns a pthread_t. Technically opaque, but an integer on linux |
| 21:39 | < Rhamphoryncus> | Or it might be a struct containing an integer.. can't recall |
| 21:39 | < Rhamphoryncus> | Probably another platform that does the struct |
| 21:40 | < Derakon[work]> | pthread_self is a builtin? |
| 21:40 | < Rhamphoryncus> | no, C function |
| 21:40 | < Derakon[work]> | Oh, d'oh. |
| 21:41 | < Derakon[work]> | I haven't yet figured out how the C++ and Python interact here, not entirely anyway. |
| 21:41 | < Derakon[work]> | I know where Python gets started up, but it's not clear to me how it gets control of the program flow yet. |
| 21:43 | < Derakon[work]> | So is the suggestion to just print the thread ID any time I make a call out to some Python, then? |
| 21:45 | < Rhamphoryncus> | yeah |
| 21:46 | < Derakon[work]> | What's the theory here, then? That multiple Python strings are trying to access the same objects, creating inconsistent behavior? |
| 21:46 | < Rhamphoryncus> | no, you can't have *ANY* simultaneous access to *ANY* parts of python |
| 21:47 | < Derakon[work]> | ...so shouldn't calling out to Python, by default, fire up the GIL? |
| 21:47 | < Derakon[work]> | Or would that break garbage collection &c? |
| 21:48 | < Rhamphoryncus> | If it's a python app with a C++ library, and python creates the threads, yes |
| 21:48 | < Rhamphoryncus> | if it's a C++ creating the threads.. depends on if you register them properly |
| 21:48 | < Derakon[work]> | No guarantees that anything is done properly here. |
| 21:49 | < Derakon[work]> | So your working hypothesis, then, is that thread A calls a Python function, then thread B comes along and calls a different Python function, then A and B's Python code clobber each other, causing a crash. |
| 21:49 | < Rhamphoryncus> | yup |
| 21:50 | < Derakon[work]> | Okay, thanks. *notes that down* |
| 21:50 | < Rhamphoryncus> | Python's threading API is pretty ugly at the best of times |
| 21:51 | < Derakon[work]> | Blah. |
| 21:51 | <@McMartin> | Global processing lock! |
| 21:51 | < Derakon[work]> | Oh, that's what "GPL" stands for~! |
| 21:52 | < Derakon[work]> | I assume I can trigger the GIL myself? |
| 21:52 | < Rhamphoryncus> | yes. There's several APIs in fact, and a couple of them work :P |
| 21:52 | < Derakon[work]> | Har. |
| 21:53 | < Rhamphoryncus> | Okay, PyEval_SaveThread and PyEval_RestoreThread are what you'd normally use. They do the GIL and also set the active thread state |
| 21:54 | < Rhamphoryncus> | You still need to register your threads properly though |
| 21:54 | < Derakon[work]> | These are "WX threads"...dunno if I have that much control. But thanks. |
| 21:56 | | * Rhamphoryncus browses the source.. ahh memories |
| 21:56 | < Rhamphoryncus> | PyThreadState_New for that particular thread |
| 21:57 | < Rhamphoryncus> | then pass to PyEval_Restore |
| 21:58 | < Rhamphoryncus> | When you're done and gonna kill the thread you call PyThreadState_Clear, PyEval_SaveThread, PyThreadState_Delete. Probably |
| 22:03 | < Derakon[work]> | Great, thanks. |
| 22:03 | < Derakon[work]> | So I could try just blindly calling those in the thread constructors and destructors... |
| 22:04 | < Rhamphoryncus> | http://docs.python.org/c-api/init.html |
| 22:05 | < Rhamphoryncus> | oh, oops, there's another better API for this |
| 22:05 | < Rhamphoryncus> | PyGILState_Ensure/Release |
| 22:06 | < Rhamphoryncus> | and PyEval_InitThreads is what you need to call once from the main thread, to turn everything on |
| 22:06 | < Rhamphoryncus> | the Ensure api actually supports nesting in a single thread I believe |
| 22:07 | | * Derakon[work] nods. |
| 22:07 | < Derakon[work]> | Okay, so current plan: find where we make calls that touch Python, hit the GIL on them, see if that fixes things. |
| 22:08 | < Rhamphoryncus> | yeah |
| 22:11 | < Rhamphoryncus> | Or an assertion that it's always called from the main thread |
| 22:12 | <@Consul> | I'd really like a mind-reading helmet where I can just imagine how I want a program to look and run and the computer just writes it all for me. |
| 22:12 | < Derakon[work]> | Har. |
| 22:13 | <@Consul> | Then we can use it to finally prove to middle managers that they really don't ever know what they really want a program to do. |
| 22:14 | < Rhamphoryncus> | lol |
| 22:14 | < Rhamphoryncus> | Make it for art first |
| 22:14 | <@Consul> | "Wow, you have a magic mind-reading helmet and you still don't have a program that works worth shit! Amazing." |
| 22:21 | < Derakon[work]> | This google search turns up no results: |
| 22:21 | < Derakon[work]> | AuiNotebook setnormalfont attributeerror -"This file was created automatically" |
| 22:26 | | Vornicus [~vorn@ServicesOp.Nightstar.Net] has joined #code |
| 22:26 | | mode/#code [+o Vornicus] by ChanServ |
| 22:27 | < Derakon[work]> | Heyo, Vorn. |
| 22:27 | | * Vornicus wavels |
| 22:46 | | * Derakon[work] installs the Python 2.6 binary, discovers that /usr/bin/python is still 2.5.1. "So what good did that do me?" |
| 22:50 | < Derakon[work]> | I only installed Python 2.6 so that Numpy would install without having to use the binaries, but now I have Numpy installed on 2.6 and nothing else, so I can either transfer all of my other dependencies over to 2.6, or try to transfer Numpy to 2.5. |
| 22:55 | <@McMartin> | Do the middle-managers never deal with customers? |
| 22:55 | <@McMartin> | "We need X!" "Here is X." "This is X! It is completely inadequate for our needs. We need Y." |
| 22:56 | < Derakon[work]> | Yeah, but unlike customers, middle managers are always right~ |
| 22:56 | <@Consul> | McMartin: My big fear is that a middle manager would use my magic helmet to build an interface, and it would look exactly like the enterpriseware shit we always see. |
| 22:56 | <@Consul> | Thus giving us an insight into the workings of their minds that we never really wanted to see. |
| 22:57 | | * Derakon[work] downloads the scipy source tarball, runs "python setup.py build", gets the error "AssertionError: hmm, why I am not inside SVN tree???" |
| 22:57 | < Derakon[work]> | The point of source tarballs is to be downloadable without using SVN! |
| 22:58 | <@McMartin> | Managing 10,000 seats at once is really a massive pain in the ass. |
| 22:58 | <@McMartin> | If we had a good way of doing it, it should have won by now~ |
| 22:59 | <@McMartin> | Basically, same problem, different domain~ |
| 22:59 | <@McMartin> | That said, fast iteration with a trusted, large client seems to help on that. |
| 23:00 | | * Derakon[work] eyes Google. "Apparently they use some SVN metadata to determine the version of Scipy being built. BRILLIANT." |
| 23:00 | <@TheWatcher> | ... |
| 23:04 | < Derakon[work]> | Hrm, and the binary download for Scipy that claims to be for Python 2.5 installed somewhere other than where it was supposed to. I have no idea where the library files actually are. |
| 23:06 | < gninsomniac> | "If Carmen San Diego and Waldo ever got together, their offspring would probably just be completely invisible." |
| 23:07 | <@Consul> | As it turns out, Waldo had testified against the mob. |
| 23:09 | <@McMartin> | gninsomniac: There's a comic strip to that effect at one point |
| 23:09 | <@McMartin> | Though the identity of the parents was the big reveal |
| 23:15 | < Derakon[work]> | Ahh, scipy is installing to /Library/Frameworks/Python/... instead of /System/Library/Frameworks/Python/... |
| 23:15 | < Rhamphoryncus> | Derakon[work]: usually /usr/bin/python2.6 or /usr/local/bin/python2.6 |
| 23:16 | < Rhamphoryncus> | /usr/bin/python is the domain of the OS. Changing it can break things badly |
| 23:16 | | * Derakon[work] nods. |
| 23:16 | < Rhamphoryncus> | Your app should be explicitly using the one it wants |
| 23:18 | < Derakon[work]> | Yeah, I got that. |
| 23:18 | < Derakon[work]> | I'm wrestling with dependency install locations ATM. |
| 23:18 | < Derakon[work]> | I hate dealing with this shit... |
| 23:19 | < Derakon[work]> | ...oh, shit. |
| 23:19 | < gninsomniac> | Hmm. To attempt to sleep or to write a bloom shader, that is the question. |
| 23:19 | < Derakon[work]> | I just did "sudo ln -sf site-packages/ * ." |
| 23:19 | < Derakon[work]> | So now everything's a symlink to itself. |
| 23:19 | < Derakon[work]> | Iiii just blew away all my Python libraries. |
| 23:19 | | ToxicFrog [~ToxicFrog@Admin.Nightstar.Net] has joined #code |
| 23:19 | | mode/#code [+o ToxicFrog] by ChanServ |
| 23:20 | < Derakon[work]> | TF, I fucked up. |
| 23:20 | < Derakon[work]> | I just did "sudo ln -sf site-packages/ * ." |
| 23:20 | < Derakon[work]> | While in the directory of Python library files. |
| 23:21 | < Rhamphoryncus> | Derakon[work]: ouch! |
| 23:21 | < Rhamphoryncus> | what OS/ |
| 23:21 | <@TheWatcher> | Oh shit |
| 23:22 | < Derakon[work]> | Rhamph: OSX. |
| 23:22 | | * Derakon[work] seeks the official Apple Python install, cannot find it. |
| 23:22 | < Rhamphoryncus> | x_x |
| 23:23 | <@TheWatcher> | which version of OSX? |
| 23:23 | < Derakon[work]> | 10.5.something |
| 23:23 | <@TheWatcher> | Hm. |
| 23:23 | <@McMartin> | Probably 10.5.8 |
| 23:23 | <@TheWatcher> | Myst might have the appropriate data on her mac if you can't locate it - I could probably tarball it for you |
| 23:23 | < Derakon[work]> | 10.5.4, actually. |
| 23:23 | < Derakon[work]> | I'm lazy about updates. |
| 23:24 | < Derakon[work]> | TW: thanks, it may come to that. |
| 23:24 | <@TheWatcher> | *nod* if it does, just tell me which bits you need. |
| 23:25 | < Derakon[work]> | Everything in /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5 |
| 23:29 | | * Derakon[work] compares the contents of the 2.3 libraries on his desktop computer (running OSX 10.4). |
| 23:29 | < Derakon[work]> | Looks like I'm mostly just missing stuff I installed for Jetblade and Fusillade. |
| 23:30 | < Derakon[work]> | But I've no idea if the 2.3 libs are compatible. |
| 23:30 | <@TheWatcher> | packing up now, dera |
| 23:30 | < Derakon[work]> | Awesome, thanks. |
| 23:32 | < Derakon[work]> | Man, it's been ages since I last shot myself in the foot with sudo. |
| 23:32 | <@ToxicFrog> | Oh dear |
| 23:33 | <@TheWatcher> | Dera: http://fleet.starforge.co.uk/python2.5.tar.bz2 |
| 23:33 | < Derakon[work]> | Downloading. 4 minutes to go. |
| 23:33 | <@TheWatcher> | (nearly 8mb, so it might take a few) |
| 23:37 | | * Vornicus has not yet managed to shoot himself in the foot with sudo. |
| 23:37 | < Derakon[work]> | I in fact can't remember my last sudo mistake, so this may be my first. |
| 23:37 | < Derakon[work]> | My roommate in college once did "rm -rf * ~" though. |
| 23:38 | < Derakon[work]> | Got it! |
| 23:39 | <@Vornicus> | wheeee |
| 23:41 | < Derakon[work]> | Er, TW...were there any .py files in that directory? |
| 23:41 | < Derakon[work]> | All I have here are .pyc and .pyo files. |
| 23:41 | <@TheWatcher> | That's all there was in there |
| 23:42 | < Derakon[work]> | Hmm... |
| 23:42 | < Derakon[work]> | Well, the .pyc files should be sufficient... |
| 23:42 | < Derakon[work]> | Maybe they removed the .py files to save space. |
| 23:42 | < Derakon[work]> | Much thanks, anyway. |
| 23:43 | <@TheWatcher> | Hope it helps. I can do more digging if you need. |
| 23:46 | < Derakon[work]> | Roger. |
| 23:55 | <@Vornicus> | .pycs are enough, generally. |
| 23:55 | <@Vornicus> | I got caught out by that when I was working on bisect: running the backcompat tests when I had my new bisect.pyc in . didn't work. :) |
| 23:57 | < Derakon[work]> | And Rhamphoryncus had some problems with my Cython code because he still had a copy of platform.pyc in his local repo after I'd removed platform.py. |
| 23:57 | | gninsomniac [lenin@Nightstar-1382.A163.priv.bahnhof.se] has quit [Quit: Z?] |
| 23:57 | < Derakon[work]> | (Which had to go because "platform" was meant for something else under Cython) |
| --- Log closed Sat Sep 05 00:00:11 2009 |