code logs -> 2018 -> Wed, 19 Dec 2018< code.20181218.log - code.20181220.log >
--- Log opened Wed Dec 19 00:00:11 2018
--- Day changed Wed Dec 19 2018
00:00
<@Reiv>
So I have decided to harken back to the ancient train systems of yore, and am attempting to implement a baton system.
00:00
< Mahal>
Okay
00:00
<&ToxicFrog>
Reiv: ok, so why not have attunity mirror the files into a directory *next* to the one the SQL server imports from, and then run something on *that* machine that watches for them, edits them, and feeds them to SQL?
00:00
< Mahal>
how are the output files named?
00:01
< Mahal>
Or that
00:01
<@Reiv>
So the ETL process dumps out not only the file, but records when the file successfully kicked off, so we know when to run the deltas from /next/ time. This gets passed to our SQL instance, who noms the file and throws the baton back to the ETL over the same shonky mirroring system
00:01
<@Reiv>
... so that no matter how desync'd things actually get in "The ETL process is running twenty minutes late and Attunity hasn't run for the last half hour", or whatever, the system at least knows what timeframe the most-recently-known-successfully-recieved-timestamp was.
00:02
<@Reiv>
Output files can be named however we want, however, Attunity does not know how to tidy a file directory; it is expected if a file is updated it is overwriting the same file name.
00:02
< Mahal>
Yeah, I'm just thinking some sort of filename with the timestamp inclusive
00:02
<@Reiv>
For clarity, this is Option 4 of an Increasingly Bad Suite Of Options here
00:02
< Mahal>
and parse that as part of your code
00:03
<@Reiv>
Option 1: Baton file. Failure state: Baton file can be passed without actually passing tables with it, and vice versa.
00:04
<@Reiv>
Option 2: File-level batons, stored in file name. Failure state: Attunity has no idea this is a new copy of an old file, and will agressively defend the recipient directory from having old files deleted. We cannot order it to delete a file in the source directory short of running a seperate cleanup script independently of the ETL process, and askin
00:04
<@Reiv>
g the server team Really Nicely to let it run for us.
00:04
<&[R]>
Does this "baton system" have another name?
00:04 * [R] is trying to research that
00:04
<@Reiv>
Option 3: File-level batons, stored inside the file. Failure state: Fuck me we just made our .csv files really fuckin' messy didn't we
00:05
<&ToxicFrog>
[R]: token passing?
00:05
<@Reiv>
[R]: Oh, probably, I ended up using batons as a metaphor because my boss doesn't speak very good english but whose father helped run the South African rail network, so 'rail baton' was a concept he could seize upon when everything else was failing horribly~
00:05
< Mahal>
HOnestly, this entire system needs to be taken out the back and shot
00:05
<&ToxicFrog>
^
00:05
<@Reiv>
Yes, yes it does
00:05
< Mahal>
Including Attunity, because there are BUILT IN WINDOWS FEATURES that do all of this
00:06
<@Reiv>
Wait til I tell you the part that this is the /upgrade/ we are spending six figure sums moving to!
00:06
< Mahal>
speaking from the very personal experience of *running them*
00:06
<@Reiv>
And I'm handcrafting a fucking bespoke ETL mirroring tool
00:06
<@Reiv>
And I'm doing it because as part of the /upgrade/ they remove our ability to access the database once it's in the cloud under any circumstances
00:07
<@Reiv>
So if we want, oh, I dunno, table data so we can /synchronise with our other systems/ we have to drag it out of the cloud through their shonky frontend, via Attunity, and then make do with what we get on a hilariously unreliable schedule.
00:08
<&ToxicFrog>
Why is this "upgrade" removing your ability to query the database you need
00:08
<&ToxicFrog>
What the hell
00:08
<@Reiv>
Because we're Moving To The Vendor's Proprietary And Super Awesome Fancy Cloud(tm).
00:09
<&McMartin>
Our cloud will block out the sun
00:09
<@Reiv>
In which we no longer need access to the database system, because they leverage the synergistic power of scalable perfomance models to ... I can't keep up the bullshit long enough, sorry, but it boils down to "Our engineers do everything for everyone and we don't want anyone customising anything, so we lock down read-only access while we're at it a
00:09
<@Reiv>
nd call it security"
00:10
<&[R]>
<ToxicFrog> Why is this "upgrade" removing your ability to query the database you need <-- Vendor Lock-In is a valuable user-oriented feature set
00:10
<@Reiv>
Their answer to "But what about the systems that needed to integrate with your data" is "Oh, we have modules that can do that same functionality, we reccomend you purchase liscencing for the new solution"
00:10
<@Reiv>
It is impressive how point-blank they will smile while they tell us this
00:11
<@Reiv>
It's even better when they don't even flinch when they state that some of these modules are 'on the roadmap for late 2019'
00:11
<&McMartin>
Why are these vendors not being politely shown the door
00:11
<&McMartin>
... possibly to the scorpion pit
00:11
<@Reiv>
Because they're one of the market leaders
00:11
<&McMartin>
Though I do acknowledge that part of sales is that the customer is the one who has to tell you that this isn't a fit
00:12
<&McMartin>
But they don't solve your problem, so why are you buying from them, etc
00:12
<@Reiv>
We changed /to/ them about a decade ago, judging by the data artefacts in the DB
00:12
<@Reiv>
And this is their Latest And Greatest Offering and our head manager is dead-set we gotta get into the cloud, and for this particular solution this /is/ their cloud
00:13
<&ToxicFrog>
So, they have an existing solution, that works, and that you have been using for ten years
00:14
<@Reiv>
And is due to stop being supported, as all future support is to be done via the cloud version only
00:14
<&ToxicFrog>
But now they have a new product which is ~in the cloud~ and your decider has Decided to "upgrade" to that whether or not it has feature parity with what you're already using?
00:14
<@Reiv>
Oh, this includes any whiff of mobility support, which is a major executive-level mandate
00:15
<&[R]>
TF: The ability to decide things, the ability to think critically, morals and ethics. Pick two.
00:15
<@Reiv>
(This is otherwise known as "Councils are going to get laughed at and/or Examined Suspiciously By Central Government, ahem, if people cannot usefully do their paperwork via the website sometime this fucking decade")
00:16
<@Reiv>
Our vendors have seized upon this Exciting Opportunity to Present Their Latest Product. Enjoy your lock-in.
00:17
<@Reiv>
And their hilarious features.
00:17
<@Tamber>
"We couldn't fix it, so now it's a feature"?
00:17
<@Reiv>
I still get a couple requests a month to delete individual rows out of the database because there's /no button in the front end to delete a mistaken consumption id/
00:18
<@Reiv>
Those will become Tech Support requests that get escalated to the AMS consulting team for vetting, before being passed onto the Engineering Team for technical support!
00:18
<@Reiv>
We, uh, do not have a contract with Engineering; no-one is supposed to need their help often enough that they even sell that as an option.
00:18 * Reiv has already got the packet of marshmellows and a really long stick for /that/ particular little bonfire.
00:19
<@Reiv>
Don't forget, however
00:19
<@Reiv>
We have a council motto, an ethos to everything we do as an organisation
00:19
<@Reiv>
It is (are you sitting down?)
00:19
<@Reiv>
"Can do, Will do."
00:19
<@Tamber>
...does that get vandalised a lot?
00:19
<@Reiv>
Pause a second, and consider what that means for an IT & Infrastructure team.
00:20
<@Tamber>
suffering, mostly?
00:20
<@Reiv>
There is a project that has become a running joke
00:20
<@Reiv>
I started it on my first month on the job here, three years ago.
00:20
<@Reiv>
It was meant to take three months.
00:21
<@Reiv>
After nine months, I started getting suspicious questions about why it wasn't done yet; I'd had a promising prototype by week six!
00:21
<@Reiv>
"Because I've already had signoff and finished the fucking project three times."
00:21
<@Reiv>
I am still working on that project.
00:21
<@Reiv>
The total is now five, and will hopefully soon be six.
00:21
<@Reiv>
Local government, ladies and gents~
00:30
<@Reiv>
So, yes
00:31 * Mahal blinks
00:31
< Mahal>
so you've finished it three times & it's been reopened with new stuf?
00:31
<@Reiv>
Every single time.
00:31
<@Reiv>
Still the same project, because it's still "The water meter project", it just had some changes!
00:31
<@Reiv>
You know. After we'd signed it off as done.
00:32
<@Reiv>
Because the councillors don't much care what status an IT dept sets its work to.
00:34 * Reiv shrugs. I suspect the system wasn't bad when it was chosen ten years ago. And, to be fair, it's /no worse/ than any of the /other/ council CRM systems out there
00:35
<@Reiv>
But they've got a lot of inertia and they know it, too :p
00:38
<~Vornicus>
oog. this drive is not healthy
00:54
<~Vornicus>
Okay. I need to stop existing operations and see if I can get a disk repair of some sort to work on it
00:59
<@Reiv>
You, uh, can't get a backup off?
00:59
<~Vornicus>
It stalls on a particular file
01:00 * Reiv hands Vornicus a double-header .csv with an age-of-steam era synchronization system attached.
01:00
<~Vornicus>
I think I can get this bit off, let's see
01:31
<@macdjord>
Reiv: How about adding 3 new columns to the CSV? So the first row is just the baton data with everything else blank, and the rest of the table is the actual data witht he baton columns blank, and then you throw away the bits you don't need after parsing.
01:41
<&McMartin>
https://twitter.com/PavelASamsonov/status/1074701053034160128
01:42
<@Reiv>
macdjord: Three new columns, with a bonus row in it? Hrm.
01:42
<@Reiv>
That might actually be harder than forcing the loader to skip headers.
01:42
<@Reiv>
But yes, it's possible. Likewise we could just add three columns and stuff the numbers in and call it done.
01:44
<@macdjord>
Reiv: To calrify: add 3 new columns for the Baton stuff. The first row has the baton data in these columns, and NULL everywhere else. Every subsequent row has NULL in those 3 columns, and actual data in the others. The file is thus a neat rectangle again, which should not break your CSV parser.
01:45
<@Reiv>
'tis possible, true
01:45
<@Reiv>
Does complicate dynamically loading the tables at the other end, but shouldn't be impossible with dynamic SQL.
01:45
<@macdjord>
After parsing, extract your baton data and do whatever you need with it. Then discard that whole row, and save the rest of the data, throwing away the always-NULL baton columns.
01:47
<@macdjord>
Alternately, a simpler version: have you tried putting the baton data at the /end/ of the file? Many parsers will just silently fill in the missing columns as NULL.
01:47
<@Reiv>
In the end we just threw together a scalar function that finds out how many columns are in the target table and outputs a string of "","",etc of appropriate length to tack onto the baton header
01:47
<@Reiv>
Yeah, tried that, SQL Server was being dumb about it so no gain there
01:49
<@macdjord>
Reiv: Ah. Yeah, that works. I'd, er, assumed it wasn't an option because of type consistency or something, otherwise you'd already have done that.
01:54
<@Reiv>
Well, the pain is that it has to be able to spit out an arbitary yet precise number of columns of "",""s
01:55
<@Reiv>
So it involved faffing with scalar functions, which /then/ involves making a support ticket to get the cloud version of the software to have it exist
01:55
<@Reiv>
So it is not ideal, but it /is/ done~
01:59
<@macdjord>
Yeah, I'd been implicitly assuming that adding baton info was being done while the data was still in /data/ form, and # of columns was therefore no furthur away than the language-appropriate version of `len()`.
02:05
<@Reiv>
nope!
02:05
<@Reiv>
The baton gets created seperate to the actual table dump, albeit having access to the same system.
02:05
<@Reiv>
This is being done in pretty much Fancy UI Timed SQL.
02:05
<@Reiv>
Thank god we have an "Append file" option.
02:07
<~Vornicus>
ok that's that stuff.
02:07
<~Vornicus>
need to poke around and be a bit more selective
02:11 mac [macdjord@Nightstar-grpbnp.mc.videotron.ca] has joined #code
02:11 mode/#code [+o mac] by ChanServ
02:14 macdjord [macdjord@Nightstar-grpbnp.mc.videotron.ca] has quit [Ping timeout: 121 seconds]
02:15 Emmy [Emmy@Nightstar-9p7hb1.direct-adsl.nl] has joined #code
02:20 Emmy [Emmy@Nightstar-9p7hb1.direct-adsl.nl] has quit [Ping timeout: 121 seconds]
02:35 Kindamoody is now known as Kindamoody[zZz]
02:35 Vorntastic [uid293981@Nightstar-6br85t.irccloud.com] has joined #code
02:35 mode/#code [+qo Vorntastic Vorntastic] by ChanServ
02:39 JustBob [justbob@Nightstar.Customer.Dissatisfaction.Administrator] has quit [Connection closed]
03:06
<&ToxicFrog>
oh no
03:06
<&ToxicFrog>
I got ancilla to send me notifications via IRC.
03:06
<&ToxicFrog>
These means they go to my phone and watch.
03:06
<&ToxicFrog>
Great for the monitoring system.
03:07
<&ToxicFrog>
However, I decided to test it by enabling the "send test messages on startup" option in smartd and then restarting it.
03:07
<&ToxicFrog>
There are tend drives attached to this system and it sends a test message for each one of them.
03:07
<&ToxicFrog>
And the "test message" is the complete SMART state of the drive.
03:08
<&ToxicFrog>
I restarted it four minutes ago and it's still PMing me.
03:12
<@mac>
ToxicFrog: Oops~
03:14
<&ToxicFrog>
It's up to drive 7 now. It should be done soon.
03:20 celmin|away is now known as celmin|sleep
03:32
< Mahal>
Ahahahahaha
03:35
<@Reiv>
bahaha
03:42
<~Vornicus>
Oh no
03:55 JustBob [justbob@ServerAdministrator.Nightstar.Net] has joined #code
03:55 mode/#code [+o JustBob] by ChanServ
03:58 Derakon[AFK] is now known as Derakon
04:43 Vornicus [Vorn@ServerAdministrator.Nightstar.Net] has quit [Ping timeout: 121 seconds]
04:55 Vornicus [Vorn@ServerAdministrator.Nightstar.Net] has joined #code
04:56 mode/#code [+qo Vornicus Vornicus] by ChanServ
05:05 Vorntastic [uid293981@Nightstar-6br85t.irccloud.com] has quit [[NS] Quit: Connection closed for inactivity]
05:11 Vornicus [Vorn@ServerAdministrator.Nightstar.Net] has quit [Connection closed]
05:14 himi [sjjf@Nightstar-1drtbs.anu.edu.au] has quit [Ping timeout: 121 seconds]
05:17 Derakon is now known as Derakon[AFK]
06:34 JustBob [justbob@Nightstar.Customer.Dissatisfaction.Administrator] has quit [[NS] Quit: This unit is entering a hibernation state.]
06:44
<&jeroud>
AoC2018 19.2 is my least favourite kind of problem. :-(
06:45
<&jeroud>
Mostly because I lack sufficient data to solve it in a general way.
06:51
<&McMartin>
I'm three days behind :/
06:53
<&jeroud>
I was wondering about that.
06:53
<&jeroud>
Days 17 and 18 are good fun.
06:54
<&jeroud>
Day 19 was inevitable. There's always one of those.
06:58
<&jeroud>
Reiv: Coming in late here without reading all the context. If the first header is stuff you put there, can you not just add some extra useless empty columns to make the numbers match?
07:00
<&Reiver>
jeroud: That was the solution, yes
07:00
<&Reiver>
The fun being dynamically counting rows and inserting the blank columns :)
07:00
<&Reiver>
Not excessively hard, but, y'know, an extra step
07:00
<&jeroud>
Oh, different column counts per file?
07:01 * McMartin reads 19.1
07:01
<&McMartin>
If 19.2 is what I think it is, they've only *really* done that once before.
07:01
<&McMartin>
But I've decided I'd rather play Paper Mario tonight.
07:01
<&jeroud>
I'm pretty sure I remember it twice.
07:02
<&McMartin>
Well, hold that thought, and once I catch up I'll compare notes.
07:02
<&jeroud>
I'm using AoC as a workday warmup. Evenings are full of BotW. :-D
07:03
<&jeroud>
#aocspoilers for notes comparing. :-)
07:04
<&McMartin>
Right
07:05
<&McMartin>
Given the general pace of the days lately I don't expect to catch up until the 22nd puzzle at the very earliest.
07:08
<&Reiver>
jeroud: Yeah, these are pretty much arbitrary files
07:09
<&jeroud>
McMartin: I found 17 and 18 rather quicker than the previous batch, but YMMV.
07:10
<&jeroud>
Reiver: Do they at least have "sensible" CSV formatting?
07:10
<&jeroud>
Last time I had to do anything with CSV I ended up dealing with multiple mutually-contradictory sets of escaping rules.
07:13
<&Reiver>
jeroud: Well, I'm still speccing it myself
07:14
<&Reiver>
But it will be at least consistent whatever I get~
07:15
<&jeroud>
Yeah. "Import user-provided CSV files full of data" is a very different problem, and one I don't wish on anyone.
07:16
<&jeroud>
Our eventual solution was "use XLS files instead". Far nastier to work with, but at least they're consistent.
07:17
<&jeroud>
We still had to "unsmarten" all the punctuation, though.
07:36
<&McMartin>
jeroud: Yeah, but I am getting Holiday'd quite hard for several days and am not sure when I'll have the spare time.
07:39
<&jeroud>
Ah, right. While those two may be quick, the following ones probably won't be.
07:50
<&McMartin>
Speaking of, I should do my initial run at packing.
08:01
<&Reiver>
Yeah, I love the csv format for being so flexible
08:01
<&Reiver>
I hate it for being so flexible
08:01
<&Reiver>
It is, at least, a format, I guess
08:02
<&Reiver>
But yeah, I've elected to set up a token-passing system so the exporting data set knows when the systems downstream have successfully consumed the data.
08:03
<&Reiver>
I could do with a better way to explain it, though.
08:04
<&Reiver>
It is the time the export system kicked off the process, only recorded if the export-from-the-system step succeeds (Yes, that step can fail too~)
08:05
<&Reiver>
The file is then passed downstream, consumed by our local SQL system, which then sends the time-token back /upstream/ to the Export System, who can then read it and go "Aha, good, you got everything up to my 9am write. I'll use that as the new start time for the next delta package"
08:05
<&Reiver>
And it can then spit things out and if the system breaks, the system'll keep using teh last-recieved-token time as the start cutoff for the new delta, which will just continually add more data to the to-be-collected files.
08:06
<&Reiver>
So if it's been generating since 9am, and then at 11am hears back that the downstream has completed 10am, the cutoff moves from 9am-til-now to 10am-til-now.
08:07
<&Reiver>
That way we don't miss stuff, even if we have pretty good odds of duplicating effort; thankfully the local SQL server /is/ under our control, so it'll be checking the delta files for duplicates of data it's already seen~
09:31 Vornicus [Vorn@ServerAdministrator.Nightstar.Net] has joined #code
09:31 mode/#code [+qo Vornicus Vornicus] by ChanServ
10:16 Kindamoody[zZz] is now known as Kindamoody
10:41 JustBob [justbob@ServerAdministrator.Nightstar.Net] has joined #code
10:42 mode/#code [+o JustBob] by ChanServ
12:05 gnolam_ [lenin@Nightstar-6ob4nc.cust.bahnhof.se] has joined #code
12:07 gnolam [lenin@Nightstar-ggccfi.cust.bahnhof.se] has quit [NickServ (RECOVER command used by gnolam_)]
12:07 gnolam_ is now known as gnolam
12:07 mode/#code [+o gnolam] by ChanServ
12:29
<&ToxicFrog>
"It is, at least, a format, I guess"
12:29
<&ToxicFrog>
That's the worst part; it isn't
12:29
<&ToxicFrog>
It's like a dozen closely related formats with no formal specification that are mostly, but not universally, compatible and all of which people just call "CSV"
12:30
<&ToxicFrog>
Bring back UNIT SEPARATOR/FIELD SEPARATOR, IMO.
12:59
<@TheWatcher>
JSON or death!~
13:26
<&ToxicFrog>
JSON also has issues but not as many as CSV
13:26
<&ToxicFrog>
I will grudgingly accept it as a replacement
13:28 celmin|sleep is now known as celmin|away
14:49 macdjord [macdjord@Nightstar-grpbnp.mc.videotron.ca] has joined #code
14:49 mode/#code [+o macdjord] by ChanServ
14:51 mac [macdjord@Nightstar-grpbnp.mc.videotron.ca] has quit [Operation timed out]
14:55 gnolam_ [lenin@Nightstar-ggccfi.cust.bahnhof.se] has joined #code
14:56 gnolam [lenin@Nightstar-6ob4nc.cust.bahnhof.se] has quit [Connection closed]
14:56 gnolam [lenin@Nightstar-ggccfi.cust.bahnhof.se] has joined #code
14:56 mode/#code [+o gnolam] by ChanServ
14:58 gnolam__ [lenin@Nightstar-ggccfi.cust.bahnhof.se] has joined #code
15:00 gnolam_ [lenin@Nightstar-ggccfi.cust.bahnhof.se] has quit [Ping timeout: 121 seconds]
15:01 gnolam [lenin@Nightstar-ggccfi.cust.bahnhof.se] has quit [Ping timeout: 121 seconds]
15:07 gnolam [lenin@Nightstar-ego6cb.cust.bahnhof.se] has joined #code
15:07 mode/#code [+o gnolam] by ChanServ
15:10 gnolam__ [lenin@Nightstar-ggccfi.cust.bahnhof.se] has quit [Ping timeout: 121 seconds]
15:42 Emmy [Emmy@Nightstar-9p7hb1.direct-adsl.nl] has joined #code
15:43 mac [macdjord@Nightstar-grpbnp.mc.videotron.ca] has joined #code
15:43 mode/#code [+o mac] by ChanServ
15:45 macdjord [macdjord@Nightstar-grpbnp.mc.videotron.ca] has quit [Ping timeout: 121 seconds]
15:50
< Emmy>
Huh. I'm working from home.
15:50
< Emmy>
what is this madness
16:23 Degi [Degi@Nightstar-ckv65d.dyn.telefonica.de] has joined #code
16:42
<~Vornicus>
Now if only handling things by index instead of pointer didn't *suck* for doing restructuring
16:45 Kindamoody is now known as Kindamoody|afk
17:56 M-E [Emmy@Nightstar-9p7hb1.direct-adsl.nl] has joined #code
17:58 Emmy [Emmy@Nightstar-9p7hb1.direct-adsl.nl] has quit [Ping timeout: 121 seconds]
18:45
<~Vornicus>
I did find a bug!
18:45
<~Vornicus>
I was using the wrong indexes for nexts on one of the mega triangles
18:51
<~Vornicus>
-- it was not the bug that is giving the failure on seed 28.
18:51 Kindamoody|afk is now known as Kindamoody
22:20 macdjord [macdjord@Nightstar-grpbnp.mc.videotron.ca] has joined #code
22:21 mode/#code [+o macdjord] by ChanServ
22:21 mac [macdjord@Nightstar-grpbnp.mc.videotron.ca] has quit [Operation timed out]
22:31
<~Vornicus>
oh hey. yesterday was the 6 month anniversary of the start of the Fey Mood
22:31
<~Vornicus>
I wish I could get more than an hour or two at a time to throw at said Fey Moon, it's very annoying
22:43
<@Reiv>
the Fey Mood?
22:43
<~Vornicus>
Vornonoi
22:57 himi [sjjf@Nightstar-1drtbs.anu.edu.au] has joined #code
22:57 mode/#code [+o himi] by ChanServ
23:30 M-E [Emmy@Nightstar-9p7hb1.direct-adsl.nl] has quit [Ping timeout: 121 seconds]
--- Log closed Thu Dec 20 00:00:12 2018
code logs -> 2018 -> Wed, 19 Dec 2018< code.20181218.log - code.20181220.log >

[ Latest log file ]