Analysing a lot of data

[NigeT]

Is there a way to get MLV to load a large number of reasonably large logs? I tried the multiple selection but after a long time, and several error pop-ups about mismatching headers (which makes little sense to me given the firmware hasn't been changed...) I got a pop-up saying it had run out of memory and couldn't load any more files and then just hung-up and wouldn't do anything.

What I need is a way to load only a subset of the races from the logs. I'm only interested in rpm, clt, advance and knock related traces - I want to plot the knock traces against rpm.

If not, is there a way to export to csv or xls so that I can try something like Excel? I don't really want to spend the time messing around with spreadsheets to do this, but it needs to be done as my tune has poor knock configuration which combined with what looks like false triggering of knock in is causing unnecessary changes to ignition timing.

Thanks,
Nigel.

[whittlebeast]

Apparently you added new logged fields from one log to the next.

Andy

[NigeT]

Thanks Andy, some of the logs might have come from the SD logging, others from ShadowDash - does that really make them incompatible?

Thanks,
Nigel.

[NigeT]

Back to the main subject of my query, is there a way to filter only the metrics you are interested in graphing this time while the file load is happening? I still find the MLV just hands up when loading a decent population of data files to do some analysis on.

[whittlebeast]

Do you have a way to post the entire log up on dropbox or some other server on the net? I happen to have a few different boxes I can test with. My bet is you are running out of memory.

Andy

[NigeT]

Hi Andy, I'm 100% sure you are right. That's why I asked about a way to filter, or subset, the logs so as to load only the metrics of interest. I'd like to do some analysis of less than 10 metrics over this large, for MLV, data set. I think I probably need a way to quilt convert the logs to a format that can be input to an analysis tool.
Thanks,
Nigel.

[NigeT]

I separated out the SD Card Logging logs and tried to load only those collected via ShadowDash. I also found a post saying how to start MLV with larger JVM memory parameters which makes some difference. Annoyingly although MLV now doesn't hang up after a couple of files it still rejects many of my logs with the very unhelpful "File Headers DO NOT Match!!!" pop up. I do not remember ever making any changes to logging configuration so why would the logging headers be any different, it's not like it's all the early files or all the late files even.
Anyway it does not look like MLV is going to work for what I want to do. Does anyone know of a way to convert Megasquirt logs to CSV or another format that can be loaded into Excel or something similar. Some way of filtering a subset of the metrics would be great. I also need to filter out the more obvious data value errors and then do some aggregation and simple statistics, standard deviations...
Many thanks,
Nigel.

[LT401Vette]

What are you trying to do?

You would have to look and see what fields are in one log and not in another to know what changed.
Common reason for change are If you changed firmware, or added GPS or HP fields.

[AdeV]

The .msl file is simply a tab-separated text file. If you change the extension from .msl to .csv, you can import it into Excel (or just plain open it in LibreOffice Calc); set the field separator to tab, and it will do the rest. Excel used to be limited to 65535 rows, not sure if that's still the case? That'd be a honking great file if it was... I don't know what limit LibreOffice Calc has, if any.

Now you can find the start timestamp you want, and delete all rows above it except for the first 4 (the headers). Find the end timestamp & you can cut off the extraneous data from the end as well. Save the file back in the same format (tab separated), and you should now be able to open it in MegaLogViewer for the pretty graphs.

If you know the start & end time stamps that you're interested in, and you can post a sample of the log file, I could probably throw together an AWK script which would strip out the data you weren't interested in.

Assuming you're running Windows... to easily get a sample of the file:

1) Download a program called BareTail (https://www.baremetalsoft.com/baretail/ - get the free version)
2) Open your MSL file in BareTail (drag & drop from explorer, or use the Open File dialog)
3) Press CTRL+HOME to go to the start of the file. Click the first line to highlight it. Scroll down a few pages, shift-click the last line to highlight everything from 1st to last line.
4) Press CTRL+C to copy it (or Edit -> Copy from the menus).
5) Open Notepad* & paste the contents. Save as a .txt file (or .msl or .csv, whichever you fancy). Upload the sample file here.


*There are better alternatives to NotePad: TextPad is my favourite. On Linux, use Pluma (but I'm not sure what you'd use in place of Baretail).


Can't help with the multiple files not loading, that does sound like the headers are different. Presumably MegaLogViewer requires identical log formats to correctly load multiple files; to fix this in Excel or similar would be a big old manual job. However, if this is something that's happening to you all of the time (and if it's happening to other people as well), drop me a PM, I might be able to throw together a program which will reformat 2 logs to match each other (obviously any missing values would be zeroed) in format.

Cheers,
Ade.

[NigeT]

Thanks Ade, great info. I feel dumb now for not digging around the file format.
I'm now playing around with some code to filter the metrics I want out of the log file and create a new file with the subset of metrics only. This should fix the headers mismatches hopefully. I'll also add a way to filter out a window of data around a date-time using the date-time in the header and the time since log started recorded for the metric samples.
Would the default logging with ShadowDash be different from that with TunerStudio? I very rarely plug the laptop in as it is too cumbersome to use while driving solo, but I guess a few of the logs could have been via TS.
Many thanks,
Nigel.

[LT401Vette]

Would the default logging with ShadowDash be different from that with TunerStudio?

Most likely yes, just because by default the Shadow Dash logs contain GPS and Accelerometer data.