"Noisy" tpsDOT

This is a forum for discussing the development and testing of alpha MS2/Extra code. Documentation
(Runs on MS2 and Microsquirt)

Moderators: jsmcortina, muythaibxr

JasonC SBB
Helpful MS/Extra'er
Posts: 120
Joined: Wed Oct 26, 2011 7:36 pm

Re: "Noisy" tpsDOT

Post by JasonC SBB »

robs wrote:I agree that Greg's graphs look good, mine do too, but a dead flat tpsDOT graph is only cosmetically better than a jiggly one. If the jiggles are all less than tpsthresh there won't be any difference in running.

That said, one thing I was forgetting when I was urging the ditching of moving anchor is that it only evaluates tpsDOT when the physical move has been above a threshold. With a simple difference/time it is possible for a jiggle to only be (say) 0.2%, but so quick that it sets a quite large tpsDOT.
It looks to me that the "moving anchor" algorithm does the same thing as the "sliding window" algorithm I proposed for an actuator (e.g. idle valve) output to prevent "shivering" of the idle air solenoid. I mentioned elsewhere that a sliding window 1-bit wide, on the A/D output signal, would be prudent, even in a very noise free environment.

Lastly, if you are oversampling (even at just 5 ms), and do 2-pole filtering at say 10 ms each, a single 0.2% change in one sample, would result in a small tpsDOT. Filtering does this.
robs
Master MS/Extra'er
Posts: 564
Joined: Sun Jan 17, 2010 4:26 pm
Location: Sydney, Australia

Re: "Noisy" tpsDOT

Post by robs »

JasonC SBB wrote:It looks to me that the "moving anchor" algorithm does the same thing as the "sliding window" algorithm I proposed for an actuator (e.g. idle valve) output to prevent "shivering" of the idle air solenoid. I mentioned elsewhere that a sliding window 1-bit wide, on the A/D output signal, would be prudent, even in a very noise free environment.

Lastly, if you are oversampling (even at just 5 ms), and do 2-pole filtering at say 10 ms each, a single 0.2% change in one sample, would result in a small tpsDOT. Filtering does this.
Yep. We've covered this -- blending will certainly reduce the slope.

I think gslender's code is actually taking the mean of ~80 samples, not 400. Still seems like overkill, but should do a number on the noise.

I do like the sound of the hardware improvements, but I'm not competent to contribute on that front.

I'll just say once more that, given how little importance TPS actually has in the MS2/Extra model, there isn't terribly much point improving its accuracy beyond a certain point. Would like to try the model you described earlier some time though, just to see how it behaved.

Have fun,

Rob.
JasonC SBB
Helpful MS/Extra'er
Posts: 120
Joined: Wed Oct 26, 2011 7:36 pm

Re: "Noisy" tpsDOT

Post by JasonC SBB »

TPSdot in my mind is for the little shot of accel fuel that fills in before the shot of fuel provided by MAPdot accel fuel arrives.
MAP needs to be fairly heavily filtered due to the IM pulsations, and the resulting lag will produce significant throttle tip-in hesitation due to the lag in MAPdot. TPSdot filtering doesn't need to be as heavy, so it can fill in.
dontz125
Super MS/Extra'er
Posts: 4186
Joined: Mon May 11, 2009 7:14 pm
Location: York, ON
Contact:

Re: "Noisy" tpsDOT

Post by dontz125 »

Don't denigrate the usefulness of what you've accomplished - there are more than a few people running AN with no MAP, and some people who are using Baro% or SD have had issues with ridiculous dTPS signals causing gross enrichening spikes.

Well done, all!
QuadraMAP Sensor Module -- PWM-to-Stepper Controller -- Dual Coil Driver
Coming soon: OctoMAP Sensor Module
TTR Ignition Systems
gslender
Super MS/Extra'er
Posts: 1072
Joined: Fri Sep 16, 2011 5:29 am
Location: Brisbane, Australia
Contact:

Re: "Noisy" tpsDOT

Post by gslender »

JasonC SBB wrote:TPSdot in my mind is for the little shot of accel fuel that fills in before the shot of fuel provided by MAPdot accel fuel arrives.
MAP needs to be fairly heavily filtered due to the IM pulsations, and the resulting lag will produce significant throttle tip-in hesitation due to the lag in MAPdot. TPSdot filtering doesn't need to be as heavy, so it can fill in.
For reference, the current available mod has both MAP over sampling and TPS over sampling that produce very smooth signals with zero response lag. I've now been able to run AE with purely TPS and EAE and are getting brilliant results with AFR staying fairly stable with almost zero lean tip in too. This is due in part to the further smoothing of the TPSdot (by a very small moving anchor) has just made the DOT values so relaible and responsive that you can align AE directly without any false triggers or lag.

I don't see any need to use MAP with AE, but given how smooth it also is, it too could be used if required (but would expect a similar moving anchor MAPdot would be needed to achieve similar results as the combined TPS oversampling and TPSdot moving anchor).

g
Mazda MX5 + MS3 Pro
DaveC
MS/Extra Newbie
Posts: 6
Joined: Sat Nov 19, 2011 8:08 am

Re: "Noisy" tpsDOT

Post by DaveC »

JasonC SBB wrote:And applying a 2-sample point simple DOT operation on 2-pole filtered data will yield the same results as the above excerpted graph from the Pavel website. The Pavel "noise robust algorithm" is effectively the same as a 2-pole "lag" filter applied before a simple 2-point differentiation. The graph above shows a rising characteristic (1-pole differentiation), then it peaks (like a mountaintop), then falls. For it to fall, there has to be 2-poles of lopass filtering.
If you say so. I have only my own experiences trying to get useful derivatives from noisy data. It seems the 2-point finite difference never gave good results, even if I smoothed the crap out of the raw data first, and then the time delay was too large anyway. I have given up using 2-point differentiation. Saving the past 5 TPS values wouldn't be that big of a deal. They'd need to be shuffled around every time a new one is acquired and then the calculation is more complicated, but it's only once every 50 ms.

But it's all academic if oversampling can get clean enough TPS values. I don't think code should be more complicated than it needs to be.
JasonC SBB
Helpful MS/Extra'er
Posts: 120
Joined: Wed Oct 26, 2011 7:36 pm

Re: "Noisy" tpsDOT

Post by JasonC SBB »

gslender wrote: For reference, the current available mod has both MAP over sampling and TPS over sampling that produce very smooth signals with zero response lag.
It's a matter of degree.
The act of sampling itself, aka, a "discrete time system", as opposed to a "continuous time system", produces an effective "response lag" of about half the sampling rate. Your new code oversamples and averages over those samples. The averaging of those samples also produces a "response lag", equal to the time chunks over which you do the averaging (5 ms?).

Your 80x oversampling plus averaging is doing pretty much the same function as an external RC filter with a ~5 ms time constant, with the sampling reduced back to the original 5ms. (plus the moving anchor filter) The system performance will be about the same.

My suggestion for an extra robust system is to install an external RC or RCRC filter. A 2-pole software "lag filter" is not necessary for your particular setup but would be prudent and useful in the general case. To benefit from a 2-pole lag filter the system must oversample by about 4x (1.25 ms sampling). It has to have a sampling rate 4x faster than the desired "response lag" time. In your case, you have 5 ms response lag time so the sampling rate has to be 1.25 ms. In my AEM I find a filter with 20 ms of response lag time to produce no driveability issues which is why I suggested 5 ms sampling rate minimum from several pages ago.

I don't see any need to use MAP with AE, but given how smooth it also is, it too could be used if required (but would expect a similar moving anchor MAPdot would be needed to achieve similar results as the combined TPS oversampling and TPSdot moving anchor).
One major use for MAPdot is to provide enrichment during rapid rise of MAP after the TPS is wide open, as a turbo spools up. It can rise at 300 kPa per second. The smoothing required on MAP to smooth out intake manifold pulsations produces a response lag that will cause leanness during turbo spoolup.
JasonC SBB
Helpful MS/Extra'er
Posts: 120
Joined: Wed Oct 26, 2011 7:36 pm

Re: "Noisy" tpsDOT

Post by JasonC SBB »

DaveC wrote:
JasonC SBB wrote:And applying a 2-sample point simple DOT operation on 2-pole filtered data will yield the same results as the above excerpted graph from the Pavel website. The Pavel "noise robust algorithm" is effectively the same as a 2-pole "lag" filter applied before a simple 2-point differentiation. The graph above shows a rising characteristic (1-pole differentiation), then it peaks (like a mountaintop), then falls. For it to fall, there has to be 2-poles of lopass filtering.
If you say so. I have only my own experiences trying to get useful derivatives from noisy data. It seems the 2-point finite difference never gave good results, even if I smoothed the crap out of the raw data first,
Using what smoothing algorithm? Some smoothing algorithms work better than others, and the one you tried may be inferior to what's inherent in "Pavel's differentiation method".
Pavel's noise robust method in one calculation does the differentiation and smoothing in one efficient set of lines of code. It appears to be a linear process such that the "principle of superposition" applies. That is, it can be separated out into smoothing and differentiation. By the principle of superposition, you can choose to do the smoothing first, *then* the differentiation. In the MS, because the smoothed data is used in different parts of the code, it makes sense to do the smoothing once, use that data in different parts of the code, then do a simple differentiation in one place.

This assumes the time constants of the smoothing is appropriate everywhere. In some places more smoothing may be necessary.

and then the time delay was too large anyway. I have given up using 2-point differentiation.
Are you comparing the time delay between algorithms that are all real time, or are you comparing real time processing with post-processing?
If the latter, you may be falling for the same trap robs fell into when he declared that the moving anchor had zero response lag - his example calculation had the error of doing post-processing instead of mimicing real time processing.
JasonC SBB
Helpful MS/Extra'er
Posts: 120
Joined: Wed Oct 26, 2011 7:36 pm

Re: "Noisy" tpsDOT

Post by JasonC SBB »

DaveC,

Here

Image

Take the first example above (N=7). What he is doing is taking some weighted average of the last 3 simple 2-point derivative calculations. That is essentially the same as doing a specific weighted lopass average of the original data, then doing a simple 2-point derivative.

The lopass that will do that is an FIR with weights = 0.2, 0.6, 0.2. It has a similar response to a 2-pole Bessel filter (Which I have been touting for a long time, and I've said 2-pole filtering must be done before any differentiation)

Below is an example with 4 columns. The 2-point differentiation after the lopass is the same as the Pavel differentiation:

raw_data; pavel sm diff N=5; my lopass; 2-point differ on my lopass;

0
0
0 0
0 0 0 0
100 20 20 20
100 60 80 60
100 20 100 20
100 0 100 0
100 0 100 0
100 0 100 0
100 0 100 0
100 0 100 0



Attached is the Excel file.

Geez, tough crows. Don't I get any credibility around here yet? This stuff has been bread and butter of my day job for the past 20 years. I'm not gonna tell Grant (who's obviously a very productive coding expert) that he's wrong about processor overhead - in the same way, give what I say some cred when it comes to analog circuits, PCB layouts, and (linear) signal filtering.
DaveC
MS/Extra Newbie
Posts: 6
Joined: Sat Nov 19, 2011 8:08 am

Re: "Noisy" tpsDOT

Post by DaveC »

Jason, I'm sorry for upsetting you. And thank you for the education.
robs
Master MS/Extra'er
Posts: 564
Joined: Sun Jan 17, 2010 4:26 pm
Location: Sydney, Australia

Re: "Noisy" tpsDOT

Post by robs »

In another thread I said I'd update my moving anchor mapDOT changes to work with the new averaging map sampling code. I have made the changes to the ms2extra_alpha_330b_20120227 version of the code.

This patch applies both the TPS and MAP moving anchor changes to the mainline code. It is still my lazy version with the parameters wired-in as constants. In building, I ran up against memory limits and uglified the code a bit to get it to fit. I have done nothing about the "bug" I think I spotted in isr_rtc.s.
tpsmappatch.zip
The car runs OK with this, though it clearly needs retuning (as you'd expect with the map sampling changes). tpsDOT and mapDOT graphs are noise free.

Have fun,

Rob.
robs
Master MS/Extra'er
Posts: 564
Joined: Sun Jan 17, 2010 4:26 pm
Location: Sydney, Australia

Re: "Noisy" tpsDOT

Post by robs »

gslender wrote:TPS over sampling that produce very smooth signals with zero response lag.
A pedantic point here, but it's not possible to have zero lag. Here are the customary graphs to demonstrate the point. Firstly the ideal around which I filled in with 100 points of Gaussian noise, then the graphs where all 100 points are averaged, and finally with only the 16 points leading up to the sampling point being averaged. 'Mean 100' is slower to respond to changes, but is more stable in its estimate of the rate of change than 'Mean 16'.
avgnoise.pdf
Considering that the objective is to detect a moving throttle, a small amount of intensive sampling near the sampling point makes more sense than blending in samples from the entire period since the last sample.

Have fun,

Rob.
gslender
Super MS/Extra'er
Posts: 1072
Joined: Fri Sep 16, 2011 5:29 am
Location: Brisbane, Australia
Contact:

Re: "Noisy" tpsDOT

Post by gslender »

robs wrote:A pedantic point here, but it's not possible to have zero lag. Here are the customary graphs to demonstrate the point. Firstly the ideal around which I filled in with 100 points of Gaussian noise, then the graphs where all 100 points are averaged, and finally with only the 16 points leading up to the sampling point being averaged. 'Mean 100' is slower to respond to changes, but is more stable in its estimate of the rate of change than 'Mean 16'.
No denying though that sampling the last N samples prior to the sample point would help (and reduce response further), and I'm looking into that now and will do some tests to see if it is worth the extra memory and cpu effort - arguably this would need to come at a cost of noise filtering as averaging two (2) samples next to each other would produce a poor result (so how much averaging is needed to reduce the noise? - and maybe this should be user configurable??).

Regardless, I think we are now splitting hairs and starting to debate things that no longer impact the outcome - of which the goal is to have smooth enough TPS and TPSdot values.

G
Mazda MX5 + MS3 Pro
JasonC SBB
Helpful MS/Extra'er
Posts: 120
Joined: Wed Oct 26, 2011 7:36 pm

Re: "Noisy" tpsDOT

Post by JasonC SBB »

Grant,

You may want to experiment with reducing the oversampling rate (perhaps down to 4x or 8x) and running a 2-pole lag filter instead of a running average. (followed by a small “moving anchor/sliding window” filter)

There must be *some* oversampling for the 2-pole lag filter to work well, and to work better than a single-pole lag factor.
The "smooth noise robust" differentiation method in the Pavel page above works well as attested to by DaveC, because it effectively applies a 2-pole lag filter along with a simple 2-point differentiation.

When setting the "lag factor" in the lag filters, the higher the sampling rate, the lower the "lag factor” % has to be to get the same resulting response lag. Oversampling means more data available to effectively "average" - that's what a lower lag factor % does. Whenever playing with different sampling rates, one must immediately change the lag factor in order to maintain the same response lag in order to do a fair comparison.

One should target an acceptable response lag when trying different filtering schemes.

Here are some numbers.

***For the case of having a sampling rate 4x the desired target of response lag***
(e.g. 2.5 ms sampling interval, 10 ms response lag)
2-pole lag filter factor must be 23%. (Note: this seems like a very slow lag factor, but remember the oversampling allows this - oversampling means more data available to effectively "average")
For the same response lag time, a single lag filter would need to have 20% lag factor, and the 2-pole lag filter would have noticeably better rejection of high frequency noise.

***For the case of having a sampling rate 8x the desired target of response lag***
(e.g. 2.5 ms sampling interval, 20 ms response lag, or 1.25 sampling interval and 10 ms response lag)
2-pole lag filter factor must be 12.2%.
For the same response lag time, a single lag filter would need to have 11.0% lag factor

For a sampling rate only 2x the desired lag filter (what I believe was tried before), there is not enough sampling to reject noise well, and the 2-pole filter shows little advantage over a single pole lag filter.

In my experience a response lag on TPSdot of 10-20 ms from a 2-pole filter works well as a compromise between noise robustness and responsiveness.

Cheers.
robs
Master MS/Extra'er
Posts: 564
Joined: Sun Jan 17, 2010 4:26 pm
Location: Sydney, Australia

Re: "Noisy" tpsDOT

Post by robs »

gslender wrote: Regardless, I think we are now splitting hairs and starting to debate things that no longer impact the outcome - of which the goal is to have smooth enough TPS and TPSdot values.
Agreed. In my defence, I did give a pedant alert. Mind you, tpsDOT was smooth enough with just the moving anchor stuff, and I'm not sure a smooth TPS buys us much.

Have fun,

Rob.
robs
Master MS/Extra'er
Posts: 564
Joined: Sun Jan 17, 2010 4:26 pm
Location: Sydney, Australia

Re: "Noisy" tpsDOT

Post by robs »

JasonC SBB wrote:Geez, tough crows
I guess you typoed "crowd", but I like the image all the same.

I don't know how you decide whether you have or haven't any credibility with this bunch of crows. You've clearly put in quite a bit of effort making and explaining both hardware and software ideas. Personally, I've had difficulty getting to grips with some of the technical language you use, but I have also put in some work trying some of them out. It's up to the core developers, Ken and James, whether they adopt these changes, so it doesn't greatly matter how much (or little) applause the flock gives you.

I'm happy being part of the unofficial "research" arm of MS R&D. You seem to fit that category. Grant too (in a more applied vein).

Have fun,

Rob.
gslender
Super MS/Extra'er
Posts: 1072
Joined: Fri Sep 16, 2011 5:29 am
Location: Brisbane, Australia
Contact:

Re: "Noisy" tpsDOT

Post by gslender »

robs wrote:..... and I'm not sure a smooth TPS buys us much.
Lower settings within the anchor, and for folks with minimal noise, no need to run an anchor at all....

G
Mazda MX5 + MS3 Pro
robs
Master MS/Extra'er
Posts: 564
Joined: Sun Jan 17, 2010 4:26 pm
Location: Sydney, Australia

Re: "Noisy" tpsDOT

Post by robs »

gslender wrote:
robs wrote:..... and I'm not sure a smooth TPS buys us much.
Lower settings within the anchor, and for folks with minimal noise, no need to run an anchor at all....

G
In buying you that, it hasn't bought you much. There are benefits to having a fairly high anchor threshold, and benefits to always running the anchor. Specifically: rejection of trivial throttle movements, even if they're real, even if they're fast.

Have fun,

Rob
JasonC SBB
Helpful MS/Extra'er
Posts: 120
Joined: Wed Oct 26, 2011 7:36 pm

Re: "Noisy" tpsDOT

Post by JasonC SBB »

I got a PM from Greg, asking why a "real" lopass (e.g. "lag" filter) is better than a simple running average.

A lopass filter is supposed to attenuate frequencies above the frequencies of interest. A running average filter doesn't increasingly attenuate higher and higher frequencies. It lets through certain higher frequencies with a "comb filter" like effect:

http://ptolemy.eecs.berkeley.edu/eecs20 ... nseRA.html

Image

In contrast a Bessel filter ("lag", or 2-pole lag in cascade) does this... here is a comparison of a running average and a Bessel fitler:

http://www.recordingblogs.com/sa/Wiki/t ... sel+window

Image
The above doesn't appear to be the same as a classical Bessel filter, but looks similar. The big difference is the classical Bessel has no ringing in response to an input impulse, the above does (click the link and the impulse response is above the pic I linked).
dontz125
Super MS/Extra'er
Posts: 4186
Joined: Mon May 11, 2009 7:14 pm
Location: York, ON
Contact:

Re: "Noisy" tpsDOT

Post by dontz125 »

I think my brain needs a filter - "lopass" keeps registering as "lop-ass" instead of "lo-pass" ... :D
QuadraMAP Sensor Module -- PWM-to-Stepper Controller -- Dual Coil Driver
Coming soon: OctoMAP Sensor Module
TTR Ignition Systems
Post Reply