Concatenating Log Files?

This is a forum for discussing the development and testing of alpha MS2/Extra code. Documentation
(Runs on MS2 and Microsquirt)

Moderators: jsmcortina, muythaibxr

Post Reply
AbeFM
Super MS/Extra'er
Posts: 875
Joined: Wed Dec 05, 2007 1:40 pm
Location: San Diego, CA
Contact:

Concatenating Log Files?

Post by AbeFM »

What does it take to string a bunch of .xls log files together from megatune? I want to autotune my maps, and I have many drives, with several different wastegate configurations, and this covers a wide range of steady-state boost values. I only tried just putting them in, and I think I ran into issues with the marks and the resets not being numbered correctly?

So, I would like to put them all together, sort out the lines that say reset/mark, and stuff them in MegaLogViewer and VE analyze. Is this reasonable? How do you do it?
2000 VVT Miata turbo, MS3Pro

Contact me if interested in a MS-II 2nd gen NB Miata PnP board.
muythaibxr
Site Admin
Posts: 8228
Joined: Thu Oct 14, 2004 12:48 pm

Post by muythaibxr »

You could probably write a quick shell script or perl script that does what you're asking...
AbeFM
Super MS/Extra'er
Posts: 875
Joined: Wed Dec 05, 2007 1:40 pm
Location: San Diego, CA
Contact:

Post by AbeFM »

But, is that the right way to do it? When I first pasted them together, I got errors, and I would like to understand them.

Ideally, a script that renumbers the resets and the marks and retains them would be best, but I don't know why it doesn't work.

But a series of 10 minute logs won't work as well as a single hour long log, for autotuning or manual sorting...
2000 VVT Miata turbo, MS3Pro

Contact me if interested in a MS-II 2nd gen NB Miata PnP board.
muythaibxr
Site Admin
Posts: 8228
Joined: Thu Oct 14, 2004 12:48 pm

Post by muythaibxr »

well, you'd probably have to figure out some way of renumbering the timestamps to be in series.

Ken
AbeFM
Super MS/Extra'er
Posts: 875
Joined: Wed Dec 05, 2007 1:40 pm
Location: San Diego, CA
Contact:

Post by AbeFM »

Ok, but there's no more "header" information that that? Just as long as "RESET" and "MARK" were in order, you're ok?

And purely for autotuning, you could just take them all out, correct?

That's good enough for most things anyway. Now I just need to con a programmer into doing it. :-)
2000 VVT Miata turbo, MS3Pro

Contact me if interested in a MS-II 2nd gen NB Miata PnP board.
Keithg
Super MS/Extra'er
Posts: 2413
Joined: Sun Mar 06, 2005 9:15 am
Location: Chicago, IL, USA
Contact:

Post by Keithg »

You could write a script or just put all of the files in Excel one after another. open one, open another copy paste, etc... Select the first 2 time variable blocks and copy to the end of the file. This should make the files look like one big file to MLV. Save as csv. It should work.

YMMV,

KeithG
AbeFM
Super MS/Extra'er
Posts: 875
Joined: Wed Dec 05, 2007 1:40 pm
Location: San Diego, CA
Contact:

Post by AbeFM »

I guess I am out of uploads, and don't know how to get rid of the old ones... So here's the source code for a script that a guy on MiataTurbo.net wrote. Took him like ten minutes and it was greatly appreciated. I'm sure it'll help someone here, too. It was the best autotune I've gotten in a long time. I had to double and redouble the memory allotment for megalog viewer, but nearly every point got touched and it's smooth too.

Code: Select all

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;

namespace MegaLogCombine
{
	class Program
	{
		static void Main( string[] args )
		{
			double timeStampOffset = 0;
			double currentTimeStamp = 0;

			int iLog = 0;
			int fieldCount = 0;

			foreach ( string arg in args )
			{
				string[] lines = File.ReadAllLines( arg, Encoding.ASCII );
				for ( int iLine = 0; iLine < lines.Length; ++iLine )
				{
					string line = lines[ iLine ];

					// strip out the first two lines of all logs besides the first
					if ( iLine < 2 )
					{
						if ( iLog == 0 )
							Console.WriteLine( line );

						continue;
					}

					// Don't want mark lines
					if ( line.Contains( "\"MARK" ) )
						continue;

					string[] fields = line.Split('\t');

					// don't do empty lines
					if ( fields == null || fields.Length < 1 )
						continue;

					if ( fieldCount == 0 )
						fieldCount = fields.Length;
					else if ( fieldCount != fields.Length )
						throw new System.FormatException( String.Format( "Found {0} fields, expected {1} fields, on line: {2}", fieldCount, fields.Length, line ) );

					double timeStamp;
					if ( !double.TryParse( fields[0], out timeStamp ) )
						throw new System.FormatException( String.Format( "Failed to parse timestamp from line {0}", line ) );

					currentTimeStamp = timeStamp + timeStampOffset;
					fields[ 0 ] = currentTimeStamp.ToString();

					StringBuilder builder = new StringBuilder();
					for ( int i = 0; i < fields.Length; ++i )
					{
						if ( i > 0 )
							builder.Append( '\t' );

						builder.Append( fields[i] );
					}

					Console.WriteLine( builder.ToString() );
				}
				timeStampOffset = currentTimeStamp;
				++iLog;
			}
		}
	}
}
2000 VVT Miata turbo, MS3Pro

Contact me if interested in a MS-II 2nd gen NB Miata PnP board.
hassmaschine
Super MS/Extra'er
Posts: 1331
Joined: Mon May 21, 2007 8:36 am

Post by hassmaschine »

that is cool - but how do I use it? (i'm not a programmer)

chris
AbeFM
Super MS/Extra'er
Posts: 875
Joined: Wed Dec 05, 2007 1:40 pm
Location: San Diego, CA
Contact:

Post by AbeFM »

Well, either you compile it, or I upload the .exe - but last time I tried to put something on here I spent so long trying to make space I gave up. Maybe someone here can compile it for you?

Or if someone knows of an attachment finder so I can delete my old attachments I'll do it.

Once installed, just do this:

MLC.exe file1.xls file2.xls file3.xls > combined.xls

that's about it. It would be nice if it did a directory, easier to set up. But it doesn't and I haven't bothered to tweak it. I just renamed all my files to 1, 2, 3, 4, 5 so it was easier to type. If you don't put in the ">" it will output to the screen, which can take like 20 minutes and then you have no file. :-) If you do it right, it takes a second or two.
2000 VVT Miata turbo, MS3Pro

Contact me if interested in a MS-II 2nd gen NB Miata PnP board.
hassmaschine
Super MS/Extra'er
Posts: 1331
Joined: Mon May 21, 2007 8:36 am

Post by hassmaschine »

AbeFM wrote:Well, either you compile it, or I upload the .exe - but last time I tried to put something on here I spent so long trying to make space I gave up. Maybe someone here can compile it for you?

Or if someone knows of an attachment finder so I can delete my old attachments I'll do it.

Once installed, just do this:

MLC.exe file1.xls file2.xls file3.xls > combined.xls

that's about it. It would be nice if it did a directory, easier to set up. But it doesn't and I haven't bothered to tweak it. I just renamed all my files to 1, 2, 3, 4, 5 so it was easier to type. If you don't put in the ">" it will output to the screen, which can take like 20 minutes and then you have no file. :-) If you do it right, it takes a second or two.
can you email it to me? I have tons of space & bandwidth, I can post it up on my website for people to download. chris AT 325ix DOT com
hassmaschine
Super MS/Extra'er
Posts: 1331
Joined: Mon May 21, 2007 8:36 am

Post by hassmaschine »

hassmaschine
Super MS/Extra'er
Posts: 1331
Joined: Mon May 21, 2007 8:36 am

Post by hassmaschine »

Post Reply