"Noisy" tpsDOT

This is a forum for discussing the development and testing of alpha MS2/Extra code. Documentation
(Runs on MS2 and Microsquirt)

Moderators: jsmcortina, muythaibxr

Post Reply
muythaibxr
Site Admin
Posts: 8228
Joined: Thu Oct 14, 2004 12:48 pm

Re: "Noisy" tpsDOT

Post by muythaibxr »

Jobro wrote: The two techniques I listed are sound DSP techniques. There will be a negligible time lag, and the consequence of having good measurements will be priceless.
I just don't think we can do those in real-time on an ms2 or ms3 chip. They require floating point calcs, which we don't really do. We might be able to get them to work with integer math, but I'd have to think about it a bit.

Ken
Megasquirt is not for use on pollution controlled vehicles. Any advice I give is for off road use only.
robs
Master MS/Extra'er
Posts: 564
Joined: Sun Jan 17, 2010 4:26 pm
Location: Sydney, Australia

Re: "Noisy" tpsDOT

Post by robs »

Jobro wrote:Hi

I meant all Raw Measurements (especially those that make a big impact of fueling or ign angle)

MAP
TP
CLT
MAT
BATTV

Fs= Sampling Frequency
Fc = Cutoff frequency

FYI meant Butterworth lowpass.

The two techniques I listed are sound DSP techniques. There will be a negligible time lag, and the consequence of having good measurements will be priceless.
Thanks for clarifying, but I think you are a bit generous in your valuation. Priceless? Really?

Let's say we get my CLT to a rock steady reading, say 84.4°C. What real difference from my reading today that wiggles around within 1°C of that value? Likewise for MAT (admittedly turbo users want this to be as responsive and accurate as possible, but that has far more to do with the sensor than noise on the line). TPS is even more trivial since its actual value isn't used for much. It would be nice to have overrun fuel cut at 0% precisely, but it's tolerable enough to have it engage at <2%. Batt.V is the one that I can see might present a tangible benefit. You want to compensate for voltage fluctuations at alternator speeds (that the injectors will actually see), but not be confused by noise at the sensor. This is tricky though because the high voltage you have just read and compensated for is just about to turn into a low voltage, so the compensation might be neatly wrong for the upcoming conditions and you might be better off just compensating for a longer term average. MAP is a particularly tricky one. As I said earlier, most of the variation in the MAP sensor reading is not noise, but the accurate variation of manifold pressure. Far more important than filtering out noise is doing something smart with the signal. The other common AD input is EGO which, once again, is variable by nature and, once again, I don't think there's much scope to improve on simple averaging of some sort.

I probably seem hypocritical here; after all, I started this thread wanting to get to the bottom of my noisy TPS readings. But I really just wanted to understand why that noise made my accel enrichment impossible to get right. I managed to find a way to get a good tpsDOT signal from amongst all the noise without needing any filter on the raw TPS reading and my accel enrichment is vastly better as a result.

Have fun,

Rob.
Philip Lochner
Super MS/Extra'er
Posts: 1004
Joined: Thu Feb 16, 2006 6:18 am
Location: George, South Africa

Re: "Noisy" tpsDOT

Post by Philip Lochner »

I can't add much to this thread other than saying that I've also noticed large values of tpsdot without supporting values of tps. I've noticed this because of ae being triggered but tps did not show reason for tpsdot to be at the levels where ae should be triggered.

Any development on this front would be welcomed :D
Kind regards
Philip
'74 Jensen Interceptor 440ci (EFI'ed with MS2 and wasted spark + GM 4L60e GPIO controlled - both on Extra FW)
JasonC SBB
Helpful MS/Extra'er
Posts: 120
Joined: Wed Oct 26, 2011 7:36 pm

Re: "Noisy" tpsDOT

Post by JasonC SBB »

Jobro wrote:Hi

I meant all Raw Measurements (especially those that make a big impact of fueling or ign angle)

MAP
TP
CLT
MAT
BATTV

Fs= Sampling Frequency
Fc = Cutoff frequency

FYI meant Butterworth lowpass.

The two techniques I listed are sound DSP techniques. There will be a negligible time lag, and the consequence of having good measurements will be priceless.
Late to this party but:

- you don't want a Butterworth filter for this application, you want Bessel... butterworths overshoot, bessels don't
- there is no need to oversample as long as:
-- the bandwidth of the information in the signal is less than half the sampling rate
-- and there is a properly designed analog filter on the A/D input which
---- provide the basic undersampling filtering
---- limit the bandwidth of the signal to the signal of interest (if this is met and the sampling rate is adequate the previous requirement will be met)
-------- the filter values I see in the MS schematics I see are woefully inadequate, time constants are waaay too fast

- There is no need for floating point math for something as simple as this - the A/D's are only 8 or 10 bits, 16 bit calcs are plenty - you just have to design the filters right

- there needs to be a basic understanding of the signal and noise characteristics before a proper filter can be spec'ed, and a proper spec is needed before the filter can be designed and tested

- nobody seems to have realized that the dot operation undoes 1 pole of filtering, and thus if the filter is a single-pole, such as the simple "lag filter", the noise gets back to the original size. A 2-pole filter works so much better than a 1-pole filter. After the dot operation there is still effectively one pole of filtering remaining.

Off the top of my head the only computationally intensive filtering are needed on RPM, TPS, and MAP, in that order of filtering speed.
The temperature signals are so slow that they can be measured and filtered every 1/10th of a second (IAT) and every 1/2 second (CLT).


My .02.
Greg G
Experienced MS/Extra'er
Posts: 311
Joined: Thu Mar 10, 2011 11:33 pm

Re: "Noisy" tpsDOT

Post by Greg G »

Am posting links to further reading for those like me who are interested but need to catch up on all this:

http://www.ti.com/lit/ml/sloa088/sloa088.pdf

http://www.mathworks.com/help/toolbox/s ... sself.html

If anyone has better links, please do post up!

On a related note...would it be useful to have the MAP signal have extra smoothing activated during closed loop idle? It should help stabilize the small PW changes during idle.
1996 Mazda MX-5 1.6L NA6/ Mazdaspeed M45 SC/ BSP AW Intercooler/ Maruha F-cams/ 425cc RX-8 injectors/ DIYPNP

MS2/Extra test mule :)
muythaibxr
Site Admin
Posts: 8228
Joined: Thu Oct 14, 2004 12:48 pm

Re: "Noisy" tpsDOT

Post by muythaibxr »

I'm of the opinion that the MAP signal should not really need any filtering at all (after my backported changes from MS3). Basically the MAP value is only updated in the main loop once per ignition event, so any filtering you do will reduce response an unacceptable amount in my opinion.

The TPS and RPM however I think could stand to benefit from some form of filtering other than the current lag factor method.

Ken
Megasquirt is not for use on pollution controlled vehicles. Any advice I give is for off road use only.
robs
Master MS/Extra'er
Posts: 564
Joined: Sun Jan 17, 2010 4:26 pm
Location: Sydney, Australia

Re: "Noisy" tpsDOT

Post by robs »

I have recently been giving gslender a hand with a small bit of coding and, in particular, we were looking into alternative ways of doing median filtering. I had doubts about how useful median filters would be in reality, but they pose an interesting programming problem.

When I mentioned my doubts about median filters, we got into a discussion which overlapped with this thread. I had thought that I must have explained my reasoning here fairly well, because gslender included my tpsDOT changes in one of his patches. But he is still pretty doubtful about it and I thought that I should perhaps give it one more go to consolidate the reasoning.

Firstly, I must emphasise that this approach is purely for the dot variables. This is not to say that smoothing the MAP, CLT, etc. is unimportant, it's just that getting a good estimate of the rate of change of a value is not well served by smoothing that value first. IOW, smoothed MAP and mapDOT should be independently derived from the raw MAP samples.

OK, so I have dusted off the R statistical computer package and defined a simple test case along with five DOT smoothing approaches.

The test case:

A value, say TPS is running at a level reading of 0.2, ten samples in it ramps up at a constant 0.2 units per sample until it reaches 4.0 where it stops changing. To this ideal, I have added some gaussian noise with a sigma of 0.05 units to get our simulated raw values. Here's what it looks like:
raw.png
The algorithms:
  • Raw: The rate of change is simply the change in y value.
  • Median: The median of the most recent 9 values is used in each slot.
  • Lag: A MS style lag of 50% is applied.
  • Moving Anchor: the algorithm I described in this thread. A movement of more than 0.5 units is considered significant. More than 4 samples without significant movement means we've stopped changing.
  • LSQ: My guess on what James and Ken might have in mind for this. The last five values have a line of best fit fitted. The slope of this line is used as the rate of change of the current value.
Code and results:

Here is the R code for these tests:

Code: Select all

#
# Implementation of MS lag averaging
#
lagavg<-function(a, lag=0.7)
{
	rv<-a;
	for (i in 2:length(a))
	{
		rv[i] = rv[i - 1]+(a[i] - rv[i - 1])*lag
	}
	return(data.frame(x<-1:length(rv), y<-rv))
}

#
# Implementation of the robs moving anchor algorithm
#
movanchor<-function(a, thresh=0.3, persist=4)
{
	x<-1;
	y<-a[1];
	for (i in 2:length(a))
	{
		if (abs(y[length(y)] - a[i]) > thresh)
		{
			# Significant movement
			x<-append(x,i);
			y<-append(y, a[i]);
		}
		else if (i - x[length(x)] > persist)
		{
			# Reset the anchor
			x<-append(x,i);
			y<-append(y, a[i])
		}
	}
	return(data.frame(x<-x, y<-y));
}

#
# Return the dot value for the data.frame provided
#
dotvalue<-function(f)
{
	return(c(0, diff(f$y)/diff(f$x)));
}

#
# Median filter
#
median<-function(a, win=3)
{
	#
	# We want the results of runmed preceded by this many dummies
	# since median(x) is to substitute for the last value in the
	# window, not the centre.
	#
	h<-trunc(win/2)
	x<-1:length(a);
	y<-c(rep(a[1], h),(runmed(a,win))[1:(length(a)-h)])
	return(data.frame(x<-x, y<-y));
}


#
# Least squares line of best fit
#
lsq<-function(a, win=5)
{
	rv<-a;
	#
	# Go a window at a time getting the lsq win samples at a time
	#
	for (i in win:length(a))
	{
		q<-(lsfit(0:(win-1), a[(i-win+1):i]))$coefficients[2];
		rv[i] = rv[i-1] + q;
	}
	return(data.frame(x=1:length(a), y=rv));
}

#
# Plot the values graph and its corresponding dot graph.
#
tplot<-function(ideal, orig, filt, col='black', title='')
{
	#
	# Plot the fixed data
	#
	yl<-range(c(ideal$y, filt$y))
	plot(ideal$x, ideal$y, type='l', col='grey40', main=title,
			xlab='x', ylab='y', ylim=yl)
	points(orig$x, orig$y, col='orange')
	lines(filt$x, filt$y, type='b', col=col)
	#
	# Plot the dot data
	#
	yl<-range(c(ideal$dot, filt$dot))
	plot(ideal$x, ideal$dot, type='l', col='grey40', ylim=yl,
			main=paste(title, "dot"), xlab='x', ylab='ydot')
	lines(filt$x, filt$dot, type='b', col=col)
}

set.seed(1000)
pdf('noise.pdf', paper='a4')

#
# Firstly, the template waveform.  Flat for 5, ramp for 20, flat for 5.
#
ideal<-data.frame(x=1:40, y=c(rep(2,10),2*(1:20),rep(40,10))/10)
ideal$dot<-dotvalue(ideal)
#
# Add Gaussian noise -- 30 samples, mean 0, sd 0.05
#
raw<-data.frame(x=1:40, y=ideal$y+rnorm(40,0,.05))
raw$dot<-dotvalue(raw)
#
# Median(9) of those points
#
med<-median(raw$y,9)
med$dot<-dotvalue(med)
#
# Lag average
#
lag<-lagavg(raw$y,0.7)
lag$dot<-dotvalue(lag)
#
# Moving anchor filter
#
manch<-movanchor(raw$y,0.5)
manch$dot<-dotvalue(manch)
#
# Moving window least-squares fit
#
lsqfit<-lsq(raw$y, 5)
lsqfit$dot<-dotvalue(lsqfit)

#
# Now plot the results
#
m<-par(mfrow=c(2,1))
tplot(ideal, raw, raw, 'red', 'Raw')

#plot(raw, type='b', col='orange', main='raw vs median(9)')
tplot(ideal, raw, med, col='red', title='Median')

#plot(raw, type='b', col='orange', main='raw vs lag(0.5)')
tplot(ideal, raw, lag, col='red', title='lag 50%')

#plot(raw, type='b', col='orange', main='raw vs movanchor(0.5)')
tplot(ideal, raw, manch, col='red', title='Moving Anchor')

tplot(ideal, raw, lsqfit, col='red', title='Moving Window LSQ')
#legend(0,9, c('raw','median(9)', 'lag 0.5', 'movanch 0.5'),
#fill=c('black','red','blue','orange'))
par(m)
dev.off()
And here is the document it generates:
noise.pdf
I did this as a PDF so you will be able to zoom in in your reader to get the full detail.

Just a quick explanation of the graphs. Each has a sensor values graph with a grey "ideal" line, an orange plot of the raw values and a red plot of the filtered values. There is also a sensor dot graph which shows the "ideal" dot line in grey, with the dot values returned by the current algorithm in red. This is the more important graph for dot purposes. The closer to the height and the start point, the better.

Raw gives something quite like what we're used to seeing. It gets the start and end of the acceleration event beautifully, but everything else is a jumble of noise.

Median gives us a similar result. It has greatly reduced the jitter on level running, but the accel event is as noisy as ever, just delayed by 5 samples. Woeful. I was a bit surprised to begin with, but it's really no surprise. When the samples are increasing, albeit roughly, the middle value of the last 9 will generally be 5 samples ago.

The 50% lag values are just like the raw samples with appreciable smoothing and a very slight delay.

The moving anchor has an appreciable delay, but it gets the levels pretty much spot on.

The moving window LSQ introduces a significant delay too, and eliminates most of the noise. Its general profile is very like the moving anchor approach.

So where does this get us? Nowhere much at the moment. I'd maintain that the moving anchor approach is the best of the bunch, but I'm happy to add more scenarios and/or algorithms (within reason) to the comparison, if you're interested.

Or you can run it yourself. R is free, and it's fairly easy to get started with (though pretty intricate when you get down to the details).

Have fun,

Rob.
piledriver
Super MS/Extra'er
Posts: 1678
Joined: Tue Oct 27, 2009 6:24 am
Location: Van Alstyne, Texas

Re: "Noisy" tpsDOT

Post by piledriver »

(/me claps, there should be a smiley for that)

Nicely done!
Once that routine is in the code it might prove to be of use elsewhere...?
Always doing things the hard way, MS2 sequential w/ v1.01 mainboard, LS2 coils. 80 mile/day commuter status.
Philip Lochner
Super MS/Extra'er
Posts: 1004
Joined: Thu Feb 16, 2006 6:18 am
Location: George, South Africa

Re: "Noisy" tpsDOT

Post by Philip Lochner »

One feature I have thought of requesting from my code-writing-challenged perspective is to have a TPS threshold below which AE is not triggered. If this threshold is at say 3% and TPS values jump between 0 and 1 with the occasional 2, AE is not triggered, but with TPS greater or equal to 3 (user settable) AE is triggered in the normal manner.

I noticed this when my car would stand and idle, NO MOVEMENT on TPS that I could see and log but AE indicator would flash now and then PW would jump up and engine rpm would dip as its "flooded" momentarily.

The more the throttle is open, the less this is a practical problem (but it still squirts fuel unnecessarily)

PS: This threshold could then also be used for those with setups where mechanical linkages where play exist between the TPS and the actual butterflies. I wonder who that might be??? Some with Jaguar V12's perhaps? :lol:
Kind regards
Philip
'74 Jensen Interceptor 440ci (EFI'ed with MS2 and wasted spark + GM 4L60e GPIO controlled - both on Extra FW)
muythaibxr
Site Admin
Posts: 8228
Joined: Thu Oct 14, 2004 12:48 pm

Re: "Noisy" tpsDOT

Post by muythaibxr »

I don't actually agree that the "dot" calcs should be taken from the unfiltered data and then filtered themselves:

1) This increases the amount of time spent filtering by a large margin
2) If you properly filter the base data (MAP, RPM, TPS) the "dot" data should be inherently cleaner anyway

Again, as far as MAP goes, unless we go back to sampling MAP asynchronously from the engine ignition events (i.e. just sample every xxx ms or something of that nature) I would say we're *already* filtering by virtue of the different MAP sampling algorithms, and probably if configured properly there shouldn't be a need for any further filtering.

I backported my "per-ignition-event averaging" method from MS3 where without tuning of any kind my MAP signal was significantly smoother.

This will be in the next MS2 alpha after I merge in gslender's changes (minus the filtering he's done), and I'll want some data from people comparing the current MAP sampling method to the new one.

Ken
Megasquirt is not for use on pollution controlled vehicles. Any advice I give is for off road use only.
gslender
Super MS/Extra'er
Posts: 1072
Joined: Fri Sep 16, 2011 5:29 am
Location: Brisbane, Australia
Contact:

Re: "Noisy" tpsDOT

Post by gslender »

Ken,

The PDF illustrates the problem fairly simply.

Whilst all attempts of filtering first try to emove the noise, ALL of them do so whilst introducing a delay to dot.

Only the anchor method removes noise AND with zero delay to dot.

You can't debate the data.

G
Mazda MX5 + MS3 Pro
gslender
Super MS/Extra'er
Posts: 1072
Joined: Fri Sep 16, 2011 5:29 am
Location: Brisbane, Australia
Contact:

Re: "Noisy" tpsDOT

Post by gslender »

I should add that with the new map sampling, it reduces the need for a change to mapDot, but you need to ensure lag factors are at 100, and even then, there is still some minor noise that folks using map based AE would benefit from the anchor approach.

In saying that, the new map sampling doesn't fix TPS noise, and any TPS filtering has the same dot problems, and really would benefit from the anchor method.

Same could apply to any other xxxdot that have used lag based filtering before dot calcs.

G
Mazda MX5 + MS3 Pro
muythaibxr
Site Admin
Posts: 8228
Joined: Thu Oct 14, 2004 12:48 pm

Re: "Noisy" tpsDOT

Post by muythaibxr »

gslender wrote:Ken,

The PDF illustrates the problem fairly simply.

Whilst all attempts of filtering first try to emove the noise, ALL of them do so whilst introducing a delay to dot.

Only the anchor method removes noise AND with zero delay to dot.

You can't debate the data.

G
However, the piece you're missing is "is the delay a problem?"

How quickly do we need to get the dot data? Is it worth the tradeoff in running all the extra filters to filter the dot data?

I don't think so... if you're oversampling the signal in question (we are), then you sample and filter at an interval that yields acceptable results in the "dot" calcs, and you're done.

Ken
Megasquirt is not for use on pollution controlled vehicles. Any advice I give is for off road use only.
muythaibxr
Site Admin
Posts: 8228
Joined: Thu Oct 14, 2004 12:48 pm

Re: "Noisy" tpsDOT

Post by muythaibxr »

gslender wrote:I should add that with the new map sampling, it reduces the need for a change to mapDot, but you need to ensure lag factors are at 100, and even then, there is still some minor noise that folks using map based AE would benefit from the anchor approach.
I would argue that you could still get acceptable results without filtering... that's why we have the thresholds for AE.

Also, if you try the new accel pump code I gave you to backport, it drastically reduces the need for the filtering on the "dot" calcs because the "enrichment" is only per-datapoint. It doesn't trigger a massive .2 second (or however long) pulse when it enriches. On my engine, I actually set the thresholds to 0 and tuned the curves accordingly, and enjoyed both smooth operation, and very fast response.
In saying that, the new map sampling doesn't fix TPS noise, and any TPS filtering has the same dot problems, and really would benefit from the anchor method.

Same could apply to any other xxxdot that have used lag based filtering before dot calcs.

G
I'm not suggesting we use lag-based filtering before the dot calcs. I'm suggesting that we should strive to find a way of filtering an oversampled signal such that the filtered result can be used for the dot calcs. That way you're doing a lot less work since you're filtering less, and you still get a good result. Even better still is if there are no tuning parameters.

I'm not saying I have the answer, I'm saying we need to keep looking.

Ken
Megasquirt is not for use on pollution controlled vehicles. Any advice I give is for off road use only.
Greg G
Experienced MS/Extra'er
Posts: 311
Joined: Thu Mar 10, 2011 11:33 pm

Re: "Noisy" tpsDOT

Post by Greg G »

I have to agree with what Jason said:
- there needs to be a basic understanding of the signal and noise characteristics before a proper filter can be spec'ed, and a proper spec is needed before the filter can be designed and tested
So we need to define these for RPM/TPS/MAP, so we can figure out what is acceptable. What is the sampling rate we need? How much noise is acceptable? How much lag is acceptable?

When we speak of downsides, is it the size of the code for the extra filtering that's the issue?

I'm afraid I can't help with the specifics (no background), just trying to add some direction :?

I do love the level of interest and participation we're getting in this thread though! :D
1996 Mazda MX-5 1.6L NA6/ Mazdaspeed M45 SC/ BSP AW Intercooler/ Maruha F-cams/ 425cc RX-8 injectors/ DIYPNP

MS2/Extra test mule :)
gslender
Super MS/Extra'er
Posts: 1072
Joined: Fri Sep 16, 2011 5:29 am
Location: Brisbane, Australia
Contact:

Re: "Noisy" tpsDOT

Post by gslender »

muythaibxr wrote: However, the piece you're missing is "is the delay a problem?"

How quickly do we need to get the dot data? Is it worth the tradeoff in running all the extra filters to filter the dot data?

I don't think so... if you're oversampling the signal in question (we are), then you sample and filter at an interval that yields acceptable results in the "dot" calcs, and you're done.

Ken
I think for MAP the new approach of over sampling reduces the issue - there is no doubt about that for me now. I would still say that people who rely on MAP based AE may still prefer a method of DOT calculation that 1) still offers some method of filtering to avoid any remaining AE jiggles and errors and 2) that don't incur a response delay (or at least minimise this to ensure they are getting a quick response to MAP).

Obviously the goal for AE should be to add more fuel the very moment more air is present - but I'll admit I'm unsure exactly how quick that can occur. So perhaps getting a solid foundation around the target would be sensible first (eg. air changes can occur in XX ms and the current sensors can detect this and the code ADC will report changes every YY ms so we need a filter that can remove noise within N events or select a method that doesn't discard events [like lag and median do]).

G
Mazda MX5 + MS3 Pro
gslender
Super MS/Extra'er
Posts: 1072
Joined: Fri Sep 16, 2011 5:29 am
Location: Brisbane, Australia
Contact:

Re: "Noisy" tpsDOT

Post by gslender »

muythaibxr wrote:I would argue that you could still get acceptable results without filtering... that's why we have the thresholds for AE.
Really? So with MAP or TPS set to a threshold of 60 is that sufficient to ensure that small changes in air are being accounted for by more fuel? How so?
muythaibxr wrote:Also, if you try the new accel pump code I gave you to backport, it drastically reduces the need for the filtering on the "dot" calcs because the "enrichment" is only per-datapoint. It doesn't trigger a massive .2 second (or however long) pulse when it enriches. On my engine, I actually set the thresholds to 0 and tuned the curves accordingly, and enjoyed both smooth operation, and very fast response.
Ahh, actively working on that port... :oops: ...if that's the case then perhaps all issues with MAP based AE (and TPS too I assume) will be minimised.
muythaibxr wrote:I'm not suggesting we use lag-based filtering before the dot calcs. I'm suggesting that we should strive to find a way of filtering an oversampled signal such that the filtered result can be used for the dot calcs. That way you're doing a lot less work since you're filtering less, and you still get a good result. Even better still is if there are no tuning parameters.

I'm not saying I have the answer, I'm saying we need to keep looking.
100% agree there. I think the best thing is that we are discussing options, reviewing real data and confirming approaches - the more thinking being done is always better.

G
Mazda MX5 + MS3 Pro
gslender
Super MS/Extra'er
Posts: 1072
Joined: Fri Sep 16, 2011 5:29 am
Location: Brisbane, Australia
Contact:

Re: "Noisy" tpsDOT

Post by gslender »

Greg G wrote:When we speak of downsides, is it the size of the code for the extra filtering that's the issue?
That plus speed and the delay filtering adds to signal response.

At the extreme end of the spectrum, a median filter with a large sliding window can introduce significant delay. So when the input changes from 10 to 100 it could take a few iterations along the window to respond with larger values, and this may add several ms delay before the rest of the code gets that changed value and then takes action.

Obviously this sometimes isn't an issue if you don't need to take action for several ms later... eg. if you cycle through code sections at 0.128ms and if it takes 20ms for air to travel down the intake, into the combustion chamber, for the piston to compress and then be at the best new position for a spark event to occur, then you can further delay the spark timing up to that 20ms and filtering can take as many cycles needed before that 20ms (assuming you can make a timing delay change right up to that point).

A very simple example, and the code is way more complex and involving that this... but the example is essentially valid to explain the issues.

This is also why Ken's point on oversampling negates the issue - if you can re-sample many times and average the results before the value is used, then we have a cleaner represented value with most of the noise removed - so this approach helps (but doesn't completely remove noise between samples as the averaging discards history between spark events).

G
Last edited by gslender on Wed Feb 15, 2012 5:21 pm, edited 1 time in total.
Mazda MX5 + MS3 Pro
robs
Master MS/Extra'er
Posts: 564
Joined: Sun Jan 17, 2010 4:26 pm
Location: Sydney, Australia

Re: "Noisy" tpsDOT

Post by robs »

muythaibxr wrote:I don't actually agree that the "dot" calcs should be taken from the unfiltered data and then filtered themselves:

1) This increases the amount of time spent filtering by a large margin
2) If you properly filter the base data (MAP, RPM, TPS) the "dot" data should be inherently cleaner anyway
Large margin? Have you looked at the patch file Ken (on page 1 of this thread)? It is tiny, and trivial. It does a subtraction, a comparison and, on the basis of that comparison decides whether or not to recompute tpsDOT which the current version recomputes every time. Alternatively, look at the graph. Each circle on the graph is a where tpsDOT is recomputed. All the skipped circles are where the movement has been deemed insignificant so the calculation is skipped. This is less work than the "unfiltered" version, not more.

Observation 2 is true if you can filter the base data to perfect. But the slightest deviation from perfect is magnified in the derivative. Look how erratic the dot graphs are compared with the pretty smooth sample graphs they're derived from.
muythaibxr wrote: Again, as far as MAP goes, unless we go back to sampling MAP asynchronously from the engine ignition events (i.e. just sample every xxx ms or something of that nature) I would say we're *already* filtering by virtue of the different MAP sampling algorithms, and probably if configured properly there shouldn't be a need for any further filtering.

I backported my "per-ignition-event averaging" method from MS3 where without tuning of any kind my MAP signal was significantly smoother.

This will be in the next MS2 alpha after I merge in gslender's changes (minus the filtering he's done), and I'll want some data from people comparing the current MAP sampling method to the new one.
Glad to hear about this new approach to MAP sampling. It has lots of intuitive appeal.
muythaibxr wrote: However, the piece you're missing is "is the delay a problem?"
Heh.
muythaibxr wrote: How quickly do we need to get the dot data? Is it worth the tradeoff in running all the extra filters to filter the dot data?
I've stated my position on the "tradeoff", but it's a strange argument to say that we get information too soon. The bulk of engine management is trying to predict the future from the past. TPS is about the only non-reactive input in that a TPS movement will usually lead a change in MAP by some interval. So you might argue that you can get TPS change information too soon (and I might call such an argument perverse), but you can't possibly argue that you get MAP change or RPM change information too soon. By definition they must have already happened and you need to get on and do what you need to do. You're already late.
Greg G wrote:
- there needs to be a basic understanding of the signal and noise characteristics before a proper filter can be spec'ed, and a proper spec is needed before the filter can be designed and tested
So we need to define these for RPM/TPS/MAP, so we can figure out what is acceptable. What is the sampling rate we need? How much noise is acceptable? How much lag is acceptable?
I agree, and very important with something like MAP not to mistake what is really signal for noise.
Greg G wrote: ... just trying to add some direction
Just like everyone else! :-)

Anyhow, I'm glad I made yesterday's posting because thinking about this stuff again, and possibly because I've coded it all up in R, I've had a bit of an inspiration.

Ken's always been particularly uncomfortable with the PERSIST parameter that my algorithm required (and is still there in the R code). This said that, if the sensor hadn't moved significantly for that period, it had stopped moving. Truth be told, I wasn't that keen on it myself as it would certainly add to lag before the end of an accel event was identified (as seen in yesterday's graphs). The dawning realisation came that Mr Newton had something to say that had some bearing on the matter, to the effect that a something in a uniform state of motion tends to stay in that state unless a force is applied to it. OK, not obvious I suppose but, based on our current dot value we can predict where the (say) TPS should be next sample. If, when the next sample is read, the TPS is within THRESH of that value, the tpsDOT value remains in effect. The moment it strays outside of that window, we can say that tpsDOT has changed and compute the new value. IOW, there's nothing special about a stopped throttle vs a steadily opening or closing throttle.

I have things to do today, but will code up a new variant in R and post the outcome in the near future. My only reservation is that it may not settle to a perfect 0, but I suspect it will in practice. We'll see.
muythaibxr wrote: I'm not saying I have the answer, I'm saying we need to keep looking.
I'm not sure that the answer exists. But I think the R testing environment might give us a good way to choose a best so far and I suspect the one just described may take that title, for a while.

Have fun,

Rob.
JasonC SBB
Helpful MS/Extra'er
Posts: 120
Joined: Wed Oct 26, 2011 7:36 pm

Re: "Noisy" tpsDOT

Post by JasonC SBB »

Guys, I have been a professional engineer for >20 years and one of my sub-specialties is EMI (interference) reduction and noise mitigation in power electronics (e.g. switching power supplies, motor drives). EMI is considered "voodoo" by many engineers but that is mainly due to lack of understanding of principles and lack of experience. Aside from that I have designed electro-mechanical motion control systems.

The engineering subtopics that covers the discussion in this thread are:
- signals and systems
- filters
- sampled data (digital) filters

While I'm glad to see some relatively in depth discussion, I see so many misconceptions in this thread.

That aside...

The best performing solution will have *all* these 4 things:

1- Proper wiring to the engine sensors, and PCB layout to minimize the noise entering the A/D inputs in the first place (e.g. noise from injector currents in shared ground return wires/traces)
2- Proper *analog* filtering at the A/D inputs, with a bandwidth not much larger than the signal of interest - the MS schematic RC filtering is waaay too fast
3- some kind of software non-linear filter (e.g. moving anchor, median)
4- some kind of linear filter (e.g. the "lag" filter)

Items 3 and 4 are merely to filter out any noise that got past 1 & 2. 1 & 2 should do the most filtering.

The reason for having both 3 and 4 is that they are good at filtering different types of noise.

Items 1 & 2 I have fixed and hacked in my AEM ECU. The effect on the MAP signal is dramatic, and I have shown datalogged results in miataturbo.net. I used a 2-pole Bessel filter on the board (described below). I also spec'd the filter by deciding what the acceptable delay and rise times were. (~60 ms IIRC)

robs,

Nice job on the R code experimentation. You're on the right track with using a ramp input.
There's a reason that step inputs and ramp inputs are some 2 standard inputs to show the performance of a filter.

Also, your use of gaussian noise is useful for measuring filter performance, but remember it is only one type of noise and has flat energy vs. frequency.

One must understand the characteristics of the noise entering the A/D inputs.

There may be some very narrow impulse noise (e.g. ignition), and also some narrow-frequency sinusoids (e.g. intake manifold pulsations from the intake valves, which is 100 Hz at 3000 RPM on a 4-cyl app). There is also quantization noise from the A/D conversion.

If so these should be tested as well. Good job on showing the results of the dot operation after the filter. This was a point I tried to make early on.

Here is why I think a 2-pole (2nd order) lopass filter is so useful. For a given ramp delay and step rise time, a 2-pole filter is much better at rejecting high frequency noise. If you do a dot operation on a signal that passed through a 2-pole filter, you will find that there is much less noise after the dot'ing. You should compare a simple (1-pole) lag filter to 2 identical ones in cascade, and adjust the lag factor of each one until you get the same delay as the simple one. You will find the noise after dot'ing is much better. The reason for this is that a 2-pole filter "lags" the signal twice, and the dot operation "undoes" 1 "lag". With a 1-pole filter you get back the original noise amplitude, with a 2-pole filter you get 1 remaining "pole" or "lag".

The best 2-pole filter for this application is a Bessel, because it has no overshoot. A 2-pole Bessel is simply 2 identical "lag" filters in cascade.

Re: oversampling, this only eases the requirement on the front end analog anti-aliasing filter, and shifts the frequency of the quantization noise up. It's not a big panacea.

Again, I will repeat. The characteristics of the signal, the noise, and the requirements of the filter needs to be determined first. If not, there will be much stabbing in the dark.
Post Reply