Feature request: show SD log length in TS

Testing and development of Megasquirt 3

Moderators: jsmcortina, muythaibxr

Post Reply
elaw
Super MS/Extra'er
Posts: 2926
Joined: Fri Oct 16, 2009 6:20 am
Location: Wilmington, MA

Feature request: show SD log length in TS

Post by elaw »

I know this impacts both TS and the firmware but I figure the latter is probably the harder part so I'm asking here...

Would it be possible when viewing the SD log file list within TunerStudio, to see not only the file size but the length of the actual recorded data (as either bytes or time)?

Since my setup always starts SD logging at key-on, I get a fair number of useless logs from when I power the car on to make tuning changes etc. The trick is that such logs are usually very short, so if the actual data size were shown in the list, it would be easy to identify the logs I really want to look at.
Eric Law
1990 Audi 80 quattro with AAN turbo engine: happily running on MS3+MS3X
2012 Audi A4 quattro, desperately in need of tweaking

Be alert! America needs more lerts.
jsmcortina
Site Admin
Posts: 39585
Joined: Mon May 03, 2004 1:34 am
Location: Birmingham, UK
Contact:

Re: Feature request: show SD log length in TS

Post by jsmcortina »

Technically not possible at present. When you key-off the MS3 is off, so there is no way to re-write the directory entry with a new file size.

Jamed
I can repair or upgrade Megasquirts in UK. http://www.jamesmurrayengineering.co.uk

My Success story: http://www.msextra.com/forums/viewtopic ... 04&t=34277
MSEXTRA documentation at: http://www.msextra.com/doc/index.html
New users, please read the "Forum Help Page".
elaw
Super MS/Extra'er
Posts: 2926
Joined: Fri Oct 16, 2009 6:20 am
Location: Wilmington, MA

Re: Feature request: show SD log length in TS

Post by elaw »

I wasn't thinking of resizing the files, but when they're enumerated isn't there a way to tell how much valid data each one contains?

When TS converts the files it somehow determines when logging stopped, I figured a similar technique could be used in the firmware.
Eric Law
1990 Audi 80 quattro with AAN turbo engine: happily running on MS3+MS3X
2012 Audi A4 quattro, desperately in need of tweaking

Be alert! America needs more lerts.
jsmcortina
Site Admin
Posts: 39585
Joined: Mon May 03, 2004 1:34 am
Location: Birmingham, UK
Contact:

Re: Feature request: show SD log length in TS

Post by jsmcortina »

elaw wrote:I wasn't thinking of resizing the files, but when they're enumerated isn't there a way to tell how much valid data each one contains?
Only by reading each file in full sequentially. That could take 10-15 minutes.

James
I can repair or upgrade Megasquirts in UK. http://www.jamesmurrayengineering.co.uk

My Success story: http://www.msextra.com/forums/viewtopic ... 04&t=34277
MSEXTRA documentation at: http://www.msextra.com/doc/index.html
New users, please read the "Forum Help Page".
elaw
Super MS/Extra'er
Posts: 2926
Joined: Fri Oct 16, 2009 6:20 am
Location: Wilmington, MA

Re: Feature request: show SD log length in TS

Post by elaw »

Ah, now I get it. That amount of time is a bit more than would be optimal. :lol:
Eric Law
1990 Audi 80 quattro with AAN turbo engine: happily running on MS3+MS3X
2012 Audi A4 quattro, desperately in need of tweaking

Be alert! America needs more lerts.
jsmcortina
Site Admin
Posts: 39585
Joined: Mon May 03, 2004 1:34 am
Location: Birmingham, UK
Contact:

Re: Feature request: show SD log length in TS

Post by jsmcortina »

Unfortunately yes. I know exactly what you are wanting and why though!

James
I can repair or upgrade Megasquirts in UK. http://www.jamesmurrayengineering.co.uk

My Success story: http://www.msextra.com/forums/viewtopic ... 04&t=34277
MSEXTRA documentation at: http://www.msextra.com/doc/index.html
New users, please read the "Forum Help Page".
elaw
Super MS/Extra'er
Posts: 2926
Joined: Fri Oct 16, 2009 6:20 am
Location: Wilmington, MA

Re: Feature request: show SD log length in TS

Post by elaw »

It probably wouldn't be worth the trouble but if you had random access to the files there might be a solution to that problem: a binary search.

Say for example the log file is 8000 bytes long. Read byte 8000, if it has data you know the log file is 100% full. If not, read the 4000th byte (the midpoint in the file) and see if it's populated. If it is, check the 6000th byte (3/4 point), if not, check the 2000th byte (1/4 point), and so on.

If my mental math is correct, you could determine the "fullness" of the file to within 10% by reading 5 bytes, and 10% accuracy would probably be good enough for most people.
Eric Law
1990 Audi 80 quattro with AAN turbo engine: happily running on MS3+MS3X
2012 Audi A4 quattro, desperately in need of tweaking

Be alert! America needs more lerts.
racingmini_mtl
Super MS/Extra'er
Posts: 9128
Joined: Sun May 02, 2004 6:51 am
Location: Quebec, Canada
Contact:

Re: Feature request: show SD log length in TS

Post by racingmini_mtl »

Actually, 5 reads in a binary search would put you closer to 3% (1/32).

Jean
jbperf.com Main site . . . . . . . . . . . . . . . . . . . . . . jbperf.com Forum
Image
elaw
Super MS/Extra'er
Posts: 2926
Joined: Fri Oct 16, 2009 6:20 am
Location: Wilmington, MA

Re: Feature request: show SD log length in TS

Post by elaw »

Good point!

I realized afterward the approach I outlined above has one "flaw" that in some circumstances would be helpful and in others not.

For a true binary search, the first read would not be at the endpoint of the file, it would be at the midpoint and you'd go from there. But reading the last byte first would enable you to quickly identify full files without having to do any more reads on those files. So if you have an abundance of full log files it would speed things up, if most of your logs are partial it would slow things down.
Eric Law
1990 Audi 80 quattro with AAN turbo engine: happily running on MS3+MS3X
2012 Audi A4 quattro, desperately in need of tweaking

Be alert! America needs more lerts.
jsmcortina
Site Admin
Posts: 39585
Joined: Mon May 03, 2004 1:34 am
Location: Birmingham, UK
Contact:

Re: Feature request: show SD log length in TS

Post by jsmcortina »

There's no per-byte random access on SDcards, you have to read in full sectors of 512 bytes.

James
I can repair or upgrade Megasquirts in UK. http://www.jamesmurrayengineering.co.uk

My Success story: http://www.msextra.com/forums/viewtopic ... 04&t=34277
MSEXTRA documentation at: http://www.msextra.com/doc/index.html
New users, please read the "Forum Help Page".
Reverant
Super MS/Extra'er
Posts: 1233
Joined: Sat Apr 15, 2006 12:39 am
Location: Athens, Greece

Re: Feature request: show SD log length in TS

Post by Reverant »

I have no knowledge of the SD card system, how hard would it be to reserve the first 10 bytes of the log and update them with the size of the log with every frame written (or every 10 frames)? This is a really neat idea (showing the actual SD log size in TS I mean) and I would love to have it during drag racing.
The man behind MS Labs
2005 Audi A3 2.0L TFSI DSG AWD - Extreme MS3
2002 Mazda Miata 1.8 6sp - Enhanced MS3 1.4.0, sequential injection, sequential ignition, big turbo, lots of boost
BigBlockMopar
Experienced MS/Extra'er
Posts: 372
Joined: Sat Dec 31, 2011 6:10 am
Location: Netherlands
Contact:

Re: Feature request: show SD log length in TS

Post by BigBlockMopar »

I'm probably overlooking a bunch of things, but isn't this is a simple calculation of:

LogDuration = TimeEnd(or LastWrittenDataTimestamp) -minus- TimeLogStart ?
Daily driver: 1973 Dodge Dart - 360ci engine - 11.3:1cr - MS3x - ignition only. 42RH/A500 OD+LU transmission / 3.23 gears
--------------
Website: https://www.bigblockmopar.nl/
YouTube Channel: https://www.youtube.com/c/BigBlockMopar
jsmcortina
Site Admin
Posts: 39585
Joined: Mon May 03, 2004 1:34 am
Location: Birmingham, UK
Contact:

Re: Feature request: show SD log length in TS

Post by jsmcortina »

BigBlockMopar wrote:I'm probably overlooking a bunch of things, but isn't this is a simple calculation of:

LogDuration = TimeEnd(or LastWrittenDataTimestamp) -minus- TimeLogStart ?
How do you determine "TimeEnd" ?
There is no "LastWrittenDataTimestamp".
Reverant wrote:how hard would it be to reserve the first 10 bytes of the log and update them with the size of the log with every frame written (or every 10 frames)?
That would halve the max log rate and greatly complicate the code.

James
I can repair or upgrade Megasquirts in UK. http://www.jamesmurrayengineering.co.uk

My Success story: http://www.msextra.com/forums/viewtopic ... 04&t=34277
MSEXTRA documentation at: http://www.msextra.com/doc/index.html
New users, please read the "Forum Help Page".
Reverant
Super MS/Extra'er
Posts: 1233
Joined: Sat Apr 15, 2006 12:39 am
Location: Athens, Greece

Re: Feature request: show SD log length in TS

Post by Reverant »

Adding a keep-alive circuit to keep the ECU powered for a couple of seconds (to write the log size in the first frame) is trivial. The same circuit would also allow us to permanently write the long term trim table to flash only on shutdown, instead of using a button. Thoughts?
The man behind MS Labs
2005 Audi A3 2.0L TFSI DSG AWD - Extreme MS3
2002 Mazda Miata 1.8 6sp - Enhanced MS3 1.4.0, sequential injection, sequential ignition, big turbo, lots of boost
jsmcortina
Site Admin
Posts: 39585
Joined: Mon May 03, 2004 1:34 am
Location: Birmingham, UK
Contact:

Re: Feature request: show SD log length in TS

Post by jsmcortina »

Reverant wrote:Adding a keep-alive circuit to keep the ECU powered for a couple of seconds (to write the log size in the first frame) is trivial. The same circuit would also allow us to permanently write the long term trim table to flash only on shutdown, instead of using a button. Thoughts?
Using the RTCC to date-stamp the files' creation time resolved the issue for me. I don't care about the reported length, just the time of the run.

James
I can repair or upgrade Megasquirts in UK. http://www.jamesmurrayengineering.co.uk

My Success story: http://www.msextra.com/forums/viewtopic ... 04&t=34277
MSEXTRA documentation at: http://www.msextra.com/doc/index.html
New users, please read the "Forum Help Page".
jamies
Master MS/Extra'er
Posts: 523
Joined: Tue Apr 07, 2009 12:22 am

Re: Feature request: show SD log length in TS

Post by jamies »

its only 2 wires to connect a datalog buttton, and one or two more if you want the led indicator, then you can choose when to log :)
elaw
Super MS/Extra'er
Posts: 2926
Joined: Fri Oct 16, 2009 6:20 am
Location: Wilmington, MA

Re: Feature request: show SD log length in TS

Post by elaw »

Yeah but for me that's kind of counter to the purpose of SD logging.

When I expect something to happen or know I want to look at some parameter(s), I'll datalog on my phone or laptop. For me, the SD log is there for when something unexpected happens, so you can examine the log afterward to see what went wrong. And since by definition unexpected things are... unexpected, I'm not going to think to press the button beforehand.

I think I've sort of fixed the issue by setting logging to trigger on RPM > 25. That should catch everything I'm interested in (I'm hoping logging starts fast enough to catch cranking), but should not create a log when I just turn the key on without starting the engine.
Eric Law
1990 Audi 80 quattro with AAN turbo engine: happily running on MS3+MS3X
2012 Audi A4 quattro, desperately in need of tweaking

Be alert! America needs more lerts.
BigBlockMopar
Experienced MS/Extra'er
Posts: 372
Joined: Sat Dec 31, 2011 6:10 am
Location: Netherlands
Contact:

Re: Feature request: show SD log length in TS

Post by BigBlockMopar »

jsmcortina wrote:
BigBlockMopar wrote:I'm probably overlooking a bunch of things, but isn't this is a simple calculation of:

LogDuration = TimeEnd(or LastWrittenDataTimestamp) -minus- TimeLogStart ?
How do you determine "TimeEnd" ?
There is no "LastWrittenDataTimestamp".

James

Doesn't every logfile have a Last Modified date given on disk?

EDIT: Nevermind, it occured to me that the MS ECU would have to have a real time clock probably.
Daily driver: 1973 Dodge Dart - 360ci engine - 11.3:1cr - MS3x - ignition only. 42RH/A500 OD+LU transmission / 3.23 gears
--------------
Website: https://www.bigblockmopar.nl/
YouTube Channel: https://www.youtube.com/c/BigBlockMopar
jsmcortina
Site Admin
Posts: 39585
Joined: Mon May 03, 2004 1:34 am
Location: Birmingham, UK
Contact:

Re: Feature request: show SD log length in TS

Post by jsmcortina »

No, there's no last modified date written. As discussed above when the ECU is off it can't write anything. Writing during the log would waste half the SPI bandwidth.

James
I can repair or upgrade Megasquirts in UK. http://www.jamesmurrayengineering.co.uk

My Success story: http://www.msextra.com/forums/viewtopic ... 04&t=34277
MSEXTRA documentation at: http://www.msextra.com/doc/index.html
New users, please read the "Forum Help Page".
Post Reply