| --- Log opened Tue Dec 18 00:00:10 2018 |
| 00:00 | | Vorntastic [uid293981@Nightstar-6br85t.irccloud.com] has quit [[NS] Quit: Connection closed for inactivity] |
| 01:05 | | Degi_ [Degi@Nightstar-807psf.dyn.telefonica.de] has joined #code |
| 01:08 | | Degi [Degi@Nightstar-ge8a3v.dyn.telefonica.de] has quit [Ping timeout: 121 seconds] |
| 01:09 | | celmin|away is now known as celticminstrel |
| 01:17 | | Degi_ [Degi@Nightstar-807psf.dyn.telefonica.de] has quit [Connection closed] |
| 01:18 | | macdjord [macdjord@Nightstar-grpbnp.mc.videotron.ca] has joined #code |
| 01:18 | | mode/#code [+o macdjord] by ChanServ |
| 01:57 | | mac [macdjord@Nightstar-grpbnp.mc.videotron.ca] has joined #code |
| 01:57 | | mode/#code [+o mac] by ChanServ |
| 02:00 | | macdjord [macdjord@Nightstar-grpbnp.mc.videotron.ca] has quit [Ping timeout: 121 seconds] |
| 02:14 | | Kindamoody is now known as Kindamoody[zZz] |
| 04:37 | | Vorntastic [uid293981@Nightstar-6br85t.irccloud.com] has joined #code |
| 04:37 | | mode/#code [+qo Vorntastic Vorntastic] by ChanServ |
| 05:05 | | Emmy [Emmy@Nightstar-9p7hb1.direct-adsl.nl] has joined #code |
| 05:11 | | Emmy [Emmy@Nightstar-9p7hb1.direct-adsl.nl] has quit [Ping timeout: 121 seconds] |
| 05:12 | | Derakon is now known as Derakon[AFK] |
| 05:21 | | himi [sjjf@Nightstar-1drtbs.anu.edu.au] has quit [Ping timeout: 121 seconds] |
| 05:24 | | celticminstrel is now known as celmin|sleep |
| 08:05 | | Kindamoody[zZz] is now known as Kindamoody |
| 08:08 | | mac is now known as macdjord|slep |
| 08:22 | | Kindamoody is now known as Kindamoody|afk |
| 08:40 | | himi [sjjf@Nightstar-v37cpe.internode.on.net] has joined #code |
| 08:40 | | mode/#code [+o himi] by ChanServ |
| 09:00 | | McMartin [mcmartin@Nightstar-rpcdbf.sntcca.sbcglobal.net] has quit [[NS] Quit: rb] |
| 09:02 | | McMartin [mcmartin@Nightstar-rpcdbf.sntcca.sbcglobal.net] has joined #code |
| 09:02 | | mode/#code [+ao McMartin McMartin] by ChanServ |
| 10:21 | | * McMartin plays with Rust's linter |
| 10:21 | <&McMartin> | This is a Hell of a linter |
| 10:21 | <@TheWatcher> | Oh? |
| 10:22 | <&McMartin> | It noticed that I had an "i = 0; for x in xs { ... i += 1; }" equivalent |
| 10:22 | <&McMartin> | And linted "hey you have a foreach and a loop variable, there's an enumerate() construct for that" |
| 10:22 | <@TheWatcher> | Huh |
| 10:23 | <&McMartin> | And the rest were things like "This code does more work than it should", "this type specification is too narrow", "you're fighting the borrow checker harder than you need to", which were less surprising but still welcome |
| 11:03 | | Emmy [Emmy@Nightstar-9p7hb1.direct-adsl.nl] has joined #code |
| 11:21 | | macdjord [macdjord@Nightstar-grpbnp.mc.videotron.ca] has joined #code |
| 11:21 | | mode/#code [+o macdjord] by ChanServ |
| 11:22 | | macdjord|slep [macdjord@Nightstar-grpbnp.mc.videotron.ca] has quit [Ping timeout: 121 seconds] |
| 12:49 | | You're now known as TheWatcher[d00m] |
| 13:58 | | celmin|sleep is now known as celmin|away |
| 14:01 | | You're now known as TheWatcher |
| 16:19 | | Kindamoody|afk is now known as Kindamoody |
| 16:34 | | gnolam [lenin@Nightstar-jf2er2.cust.bahnhof.se] has quit [Ping timeout: 121 seconds] |
| 16:35 | | gnolam [lenin@Nightstar-1sgtio.cust.bahnhof.se] has joined #code |
| 16:35 | | mode/#code [+o gnolam] by ChanServ |
| 16:36 | | gnolam_ [lenin@Nightstar-ggccfi.cust.bahnhof.se] has joined #code |
| 16:36 | | gnolam [lenin@Nightstar-1sgtio.cust.bahnhof.se] has quit [NickServ (RECOVER command used by gnolam_)] |
| 16:36 | | gnolam_ is now known as gnolam |
| 16:36 | | mode/#code [+o gnolam] by ChanServ |
| 16:39 | | gnolam_ [lenin@Nightstar-ggccfi.cust.bahnhof.se] has joined #code |
| 16:42 | | gnolam [lenin@Nightstar-ggccfi.cust.bahnhof.se] has quit [Ping timeout: 121 seconds] |
| 16:52 | | Vorntastic [uid293981@Nightstar-6br85t.irccloud.com] has quit [[NS] Quit: Connection closed for inactivity] |
| 19:41 | | gnolam_ is now known as gnolam |
| 19:41 | | mode/#code [+o gnolam] by ChanServ |
| 20:32 | | Kizor [moryok@Nightstar-e0a4sm.utu.fi] has quit [Operation timed out] |
| 20:45 | | Kizor_ [moryok@Nightstar-e0a4sm.utu.fi] has joined #code |
| 20:58 | | himi [sjjf@Nightstar-v37cpe.internode.on.net] has quit [Connection closed] |
| 21:27 | | Degi [Degi@Nightstar-807psf.dyn.telefonica.de] has joined #code |
| 22:37 | | himi [sjjf@Nightstar-1drtbs.anu.edu.au] has joined #code |
| 22:37 | | mode/#code [+o himi] by ChanServ |
| 23:07 | | * Reiv fingertappity, tries to figure out how to do a Tricky Thing in either SQL, or something SQL-adjacent |
| 23:08 | <@Reiv> | We have a CSV file. |
| 23:08 | <@Reiv> | Inside the file name we have two headers. |
| 23:08 | <@Reiv> | Opening the file you thus get something akin to |
| 23:09 | | ErikMesoy [Bruker@Nightstar-hq72t5.customer.cdi.no] has quit [[NS] Quit: Leaving.] |
| 23:09 | <@Reiv> | "FILENAME","BATON1HH:MM:SS","BATON2HH:MM:SS" |
| 23:09 | | ErikMesoy [Bruker@Nightstar-hq72t5.customer.cdi.no] has joined #code |
| 23:09 | <@Reiv> | "ColumName1","ColumnName2","ColumName3","ColumnName4","ColumName5"etc |
| 23:09 | <@Reiv> | [actual data in columns goes here) |
| 23:10 | <@Reiv> | Essentially, the first 'header' row in this CSV file is a metadata row, recording what the 'real name' of the file is, and a pair of batons that are used for resolving file synchronisation issues |
| 23:10 | <@Reiv> | The /problem/ |
| 23:10 | <@Reiv> | Is that you can tell SQL Server to load a CSV file and start reading at an arbitrary row number... |
| 23:11 | <@Reiv> | ... but it doesn't actually ignore the rows beforehand, it just doesn't commit them to the proper reading process. |
| 23:11 | <@Reiv> | So in this case, the first row it sees ... has 3 columns in it. The one afterwards most assuredly does not. |
| 23:11 | <@Reiv> | And apparently SQL Server rather expects your number of columns to remain a little more regular than that. |
| 23:13 | | Emmy [Emmy@Nightstar-9p7hb1.direct-adsl.nl] has quit [Ping timeout: 121 seconds] |
| 23:21 | | Degi [Degi@Nightstar-807psf.dyn.telefonica.de] has quit [Connection closed] |
| 23:33 | <&[R]> | sed 1d file.csv > new.file.csv; then load new.file.csv instead |
| 23:40 | <@Reiv> | and now in an enviroment where I have no commandline access~ |
| 23:41 | <@Reiv> | Which is in fact my problem, I guess, I need SQL or SQL-adjacent because I have to load this sucker through a frontend UI into enterprise software, wheee |
| 23:42 | <&[R]> | You don't have access to it on the UI side? |
| 23:42 | < Mahal> | Pardon me asking the obvious questio |
| 23:42 | < Mahal> | can you just open the csv in advance |
| 23:42 | < Mahal> | and remove the first line |
| 23:42 | < Mahal> | then load the result into your ui |
| 23:45 | <&[R]> | ^ That's what I'm getting at |
| 23:45 | <&[R]> | If not, click the following link and search for the XML solution: https://stackoverflow.com/questions/1029384/sql-bulk-insert-with-firstrow-parameter-skips-the-following-line |
| 23:45 | <&[R]> | (Where he wraps it in XML then manually parses it) |
| 23:47 | <&[R]> | Err, converts the CSV to XML then parses it |
| 23:55 | <@Reiv> | Mahal: This will be an automated process running every ~15 minutes |
| 23:55 | < Mahal> | Okay, different question. |
| 23:55 | <@Reiv> | Interesting, [R] |
| 23:55 | < Mahal> | why does the csv contain this stupid row in the first palce |
| 23:55 | < Mahal> | can it not |
| 23:55 | <@Reiv> | Because I put it there ^_^ |
| 23:56 | <@Reiv> | ;_; |
| 23:56 | <@Reiv> | The first row exists due to a /wonderful/ set of contstraints in some incredibly stupid software |
| 23:56 | <@Reiv> | We have an ETL system we are configuring to dump out table deltas as the day goes by |
| 23:56 | <@Reiv> | This is then put into a folder that is then mirrored by mirroring software that is too stupid to know when a file is mirrored; it just mirrors everything every xteen minutes |
| 23:57 | <@Reiv> | This will then get picked up by an SQL Server job on the receiving end, to then append these delta changes into the local mirror of the table |
| 23:57 | <&ToxicFrog> | What does the mirroring software run on? Could you replace it with something slightly smarter? |
| 23:57 | <@Reiv> | lolnope, enterprise provided solution |
| 23:58 | <@Reiv> | The fun part: We cannot precisely schedule the timeframe of the ETL process runs, as it simply throws the requested job onto a distributed processor queue to be done when the system decides to do it, nor can we precisely control the mirroring system beyond "Check for any file to have updated in these directories every x minutes and overwrite anythi |
| 23:58 | <@Reiv> | ng at the recieving end when it happens, unless you crash, which can happen" |
| 23:58 | < Mahal> | What *is* the mirroring solution |
| 23:58 | <&ToxicFrog> | Could you get ETL to dump into a different directory, which is then edited and moved into the mirror directory? |
| 23:59 | <@Reiv> | Attunity, if you have heard of it |
| 23:59 | < Mahal> | Yuck |
| 23:59 | <@Reiv> | The only part of this toolchain we control directly is our own local folder and the SQL server running on top of it, and yes, we'll be yoinking files when we get them but have no control over when the new ones shows up beyond that |
| --- Log closed Wed Dec 19 00:00:11 2018 |