"Noisy" tpsDOT

[Greg G]

OK, trying to bring the last post down to earth. Jason gave me the analogy of a graphic equalizer.

We want the signal to pass through but not the noise (hiss in a sound system). So we want a lopass filter to attenuate (decrease the volume) of the frequencies above the cutoff point (the hiss/noise).



Hope this helps us earthlings keep up

[JasonC SBB]

Heh, I might be the only one using "lopass" as a shortcut for "low pass".

[Greg G]

Again to put things in perspective, I think I see 2 approaches to the problem. The coders (in general, not the devs) are trying out different stuff to see what works- a more practical approach, but a bit hit and miss. The guys with EE backgrounds (Jason) approach it from a more scientific perspective. Hence the discussion of the sampling rate vs the response lag time. Given these parameters, he can decide which filter would be best. Without these parameters, it's back to hit and miss.

I realized that it does make sense to define everything (like I tried a few pages ago), when we stumbled upon the tps over sampling and the immense improvement it gave. Looking back through the thread, Jason had already suggested this quite a few pages ago! It just went over my head at the time

Being neither a coder nor an electronics engineer, I know when to leave the experts to do their stuff in their respective fields. I just ask questions to clarify and hopefully steer everything in the right direction.

Anyway, here's to more progress as we work through this. Now for my Tylenol...

[robs]

...(Jason) approach it from a more scientific perspective.

I can't let this stand. I'd be happy if you had said authoritative, or knowledgeable or experienced, but not scientific. To be scientific is to apply the scientific method: to form a hypothesis, test it, and accept or reject it accordingly. IOW, the very trial and error process that you disdainfully say the "coders" have followed is, in fact, scientific.

we stumbled upon the tps over sampling and the immense improvement it gave

Stumbled upon? Immense improvement? The extensive oversampling that Grant has implemented has indeed dramatically reduced the noise, but it is without doubt obscuring signal too with all that averaging (as the graphs I posted recently demonstrated). Reread some of the recent comments by Jason and me and you'll see that neither of us is keen on this much oversampling. My car is fine with moving anchor and no oversampling, but it might get a slightly improved accuracy of tpsDOT (and considerably more accuracy of the not very important TPS value itself) by taking the mean of 4 quick samples (or maybe the median of 3).

Perhaps my graph didn't get it across. Here it is in numbers. Imagine 100 samples which simply go from 1-100. If you average all 100 you'll get 50.5. If you average just the last 4 you'll get 98.5, a much better estimate of where the throttle has got to. That shows how much signal you can lose through excessive oversampling. The graph will be beautifully smooth, but inaccurate.

My opinion of MAP averaging is very different because MS uses MAP to estimate cylinder filling. The overall average is likely to be a good estimate of this. On the downside, mapDOT will be less responsive. For TPS, MS mostly wants to see how quickly the pedal's moving. Instantaneous samples are going to be best for this. And that's why I bias towards using TPS for big accel enrichment, and MAP for more run of the mill stuff.

hopefully steer everything in the right direction.

I think this thread has got to the point of flogging a dead horse. Grant has gone his way. I have gone my way. More to the point for the non-programmer MS user, what are Ken and James making of it?

Have fun,

Rob.

[gslender]

Perhaps my graph didn't get it across. Here it is in numbers. Imagine 100 samples which simply go from 1-100. If you average all 100 you'll get 50.5. If you average just the last 4 you'll get 98.5, a much better estimate of where the throttle has got to. That shows how much signal you can lose through excessive oversampling. The graph will be beautifully smooth, but inaccurate.

That has grossly over dramatised the imapct of oversampling on the real values seen in use. We are talking about the 80 samples taken over a 10ms period VS a single sample taken every 10ms. There is going to be a significantly different quality outcome in terms of an approximation of the needed value at that time. A single sample taken precisely every 10ms will, with no doubt, contain noise. Noise that can by 50% chance push you in the wrong direction, or further to far in the right direction (both a wrong value) - and fairly there is also a chance that this single sample is absolutely correct (for that time only).

In comparison, the oversampling ensures that the samples taken over the 10ms approximate (fairly well in fact) the average position of the throttle for that period, and continue with that level of consistency for the next period, and the next and so on...

Is that more or less wrong? Does it matter? What is the tangible difference?

My vote is that it makes zero difference - and the benefit is less noise (with less than 10ms resposne delay, and perhaps sometimes not even that much as it depends on the values sampled during that period - so it isn't a worse case 10ms error like the overdramatised example you've painted above).

I think this thread has got to the point of flogging a dead horse. Grant has gone his way. I have gone my way. More to the point for the non-programmer MS user, what are Ken and James making of it?

I've not gone my way? I've implemented both your solution and my solution and users can try either, both or neither.

I think Ken and James don't have anything to answer for ... the point of this thread and the firmware mods is to try alternatives, seek feedback and validate solutions.

So far the concrete feedback is that the oversampling works, it works much better than just Lag Factors alone, and the Moving Anchor also works too... (but obviously does nothing for TPS itself and I for one like to have my TPS values nosie free). In addition, for some users, it would seem oversampling works enough to remove the need for having Moving Anchor - such that they've been able to reduce AE thresholds and get excellent AFR results - Q.E.D

Now that is scientific yeah?

G

[Greg G]

Ah the Internet

Robs, sorry you felt offended by the words I chose. Looking back, I suppose you are correct that scientific was a poor choice of words. And then I turn around and describe the improvement as "immense." Oh the irony

To be honest I forgot to mention your mathematical approach to it, so again apologies if I offended the math guy in you.

I do think that the over sampling and your moving anchor filter work quite well together, and the eventual solution will involve them both in some form. And I do appreciate the fact that you do expound on your theories and actually implement them.

So yeah, hope we're good

Back on topic...what I would like to see is more users with different cars/engines trying all this out. More data can only help...

[Zaphod]

Hey, please don't get in a fight about your different approaches to the topic - but stay and get a "common" solution. Though I recently changed over to MS3 I read a lot on the MS2 dev board and I think you brought a lot of new input and ideas into it.
And that even has impact on the MS3 (at least I hope so).
You all have than a hell of a job with this noise topic - please don't let go!

Just my 2cents...

[robs]

I'm sorry. Just because I was a bit put out at Greg's wording didn't warrant an escalation that drew you in, Grant. At heart I think we agree that there isn't much benefit in getting things down to the nth decimal place so this is pretty much down to nitpickery.

Perhaps my graph didn't get it across. Here it is in numbers. Imagine 100 samples which simply go from 1-100. If you average all 100 you'll get 50.5. If you average just the last 4 you'll get 98.5, a much better estimate of where the throttle has got to. That shows how much signal you can lose through excessive oversampling. The graph will be beautifully smooth, but inaccurate.

That has grossly over dramatised the imapct of oversampling on the real values seen in use. We are talking about the 80 samples taken over a 10ms period VS a single sample taken every 10ms. There is going to be a significantly different quality outcome in terms of an approximation of the needed value at that time. A single sample taken precisely every 10ms will, with no doubt, contain noise. Noise that can by 50% chance push you in the wrong direction, or further to far in the right direction (both a wrong value) - and fairly there is also a chance that this single sample is absolutely correct (for that time only).

In comparison, the oversampling ensures that the samples taken over the 10ms approximate (fairly well in fact) the average position of the throttle for that period, and continue with that level of consistency for the next period, and the next and so on...

I admit I plucked the [1:100]/100 samples out of the air as something convenient, and I agree it wasn't realistic. To make it a bit realistic, move the decimal point one to the left and you get something in the right ballpark for a throttle stomp (a rate of 0->100% in .1s). So [1:10]/100 gives you an average over 100 samples of 5.05; over the last four samples it's 9.85. So the change for this sample is still only about half what it should be, and that will be reflected in this first tpsDOT value. But this only applies to the first sample for the movement. Assuming the stomp continues, the next sample will be comparing like with like and the tpsDOT will come up to the right level. As you say, is getting the correct tpsDOT 10ms later any sort of a worry? I doubt it, but the delay is definitely there.

But let's get more objective.

Benefits of oversample80:

• simple -- isr_rtc just adds the latest sample to its accumulated 32-bit sum every time; no need to test for a sampling window.
• low CPU overhead (for the same reason)
• smoothest possible

Benefits of oversample4:

• more responsive -- last four samples before 10ms will better reflect latest throttle position
• low CPU overhead -- two shifts for the divide operation. Whole accumulator will fit in 16-bit variable rather than require a 32-bit.

Not sure which "low CPU overhead" would win. I'm happy to do the assembly language changes to isr_rtc if you want to give it a go.

I've not gone my way? I've implemented both your solution and my solution and users can try either, both or neither.

That was your way. I didn't mean it as a criticism. My way was just to suit myself -- Ken didn't seem keen on my releasing my own s19 file.

I think Ken and James don't have anything to answer for ... the point of this thread and the firmware mods is to try alternatives, seek feedback and validate solutions.

Again, I wasn't suggesting they must respond. It's simply that they have control of the "blessed" MS2/Extra release and (as Zaphod highlights) the MS3 release. If they aren't enthused about oversampling they won't include it. Likewise with moving anchor.

No matter how it turns out, I have enjoyed the journey in this thread and it has definitely helped my understanding of the issues involved. I do regret how I responded to Greg's careless wording. Again, my apologies.

Have fun,

Rob.

[gslender]

If they aren't enthused about oversampling they won't include it.

Check out Alpha 19 of MS3 - it includes a TPS oversampling, but I think it is last 8 samples... so they've voted I guess ;-)

G

[Greg G]

No matter how it turns out, I have enjoyed the journey in this thread and it has definitely helped my understanding of the issues involved. I do regret how I responded to Greg's careless wording. Again, my apologies.

Have fun,

Rob.

Group hug!

[JasonC SBB]

I’d like to point out again that in theory a 2-pole (done twice) lag filter with the proper “lag %” will provide superior filtering to a simple running average filter.

The 10x / 100x / 1000x oversampled signal with running average filter obviously works very well in Greg’s and Grant’s cars, but there can be certain cars with certain types of noise that it noticeably won’t work as well as a 2-pole filter with the same or lower oversampling rate.

To use robs's example above, with a 100-sample averaging window, if the TPS changes suddenly, the last sample has the same weight as all the previous 99.
In a Bessel (i.e. "lag" filter), the last sample has much a larger effect than the one from 100 samples ago. It's heavy on math, but you can show that signals with a certain frequency (periodicity) will pass through an averaging filter, but not through a Bessel filter. And, a one-sample wide spike (impulse) will initially affect a Bessel filter more, but its effect rapidly decays away. With an averaging filter, the effect hangs around at a constant value, for the width of the whole window.

And because the lag% has a profound effect on the performance of said 2-pole filter and its resulting “response lag time”, and because the lag% isn't a meaningful number to the desired effect, I propose letting Tunerstudio calculate lag% from the desired response lag time. What I mean is, the user-entered number should NOT be the “lag %”, (which will get him completely lost) but rather the more meaningful, response lag time expressed either in milliseconds, or in # of samples. Tunerstudio should then pick the correct lag % number from an equation or from a lookup table.

For example, if the sampling rate is fixed at 1 ms, and one wants to try filtering for a 10 ms response lag time, the user will enter “10 ms” or “10 samples”.
Or if the user enters “20 ms”, then he gets heavier filtering (at the expense of slower response lag time.

I gave numbers (lag %), for the case of 8 samples lag response time, and 16 samples response lag time, in the previous page.

I propose that the 2-pole lag filter with the proper (enough) amount of oversampling and the proper "lag %" be the standard filter technique for most (all?) of the real-time signals in the MS. In my experience it's the best all around filter for these types of signals and applications, balancing simplicity, overhead, and performance (delay (response lag time), high frequency noise rejection, "rolloff", and lack of overshoots)

I've worked with filters for over a decade and even invented a very specific type of filter for a very specific type of application. In a former life I did some DSP for a satellite receiver.

For the ones who want to have fun with the math, you can look up "Laplace transforms", "Z transforms", averaging and comb filters, and Bessel filters.

[Peter Florance]

I have to weigh in here. In the cars I've tuned over the years, I have absolutely no issue with the way TPS is read on MegaSquirt's. It seems like a solution that has no problem.

Am I wrong here?

[gslender]

I have to weigh in here. In the cars I've tuned over the years, I have absolutely no issue with the way TPS is read on MegaSquirt's. It seems like a solution that has no problem.

Am I wrong here?

What AE threshold have you generally been able to manage?

G

[Peter Florance]

I have to weigh in here. In the cars I've tuned over the years, I have absolutely no issue with the way TPS is read on MegaSquirt's. It seems like a solution that has no problem.

Am I wrong here?

What AE threshold have you generally been able to manage?

G

generally around 40

If I use enough squirts in batch system, the cars are very responsive.

[robs]

I have to weigh in here. In the cars I've tuned over the years, I have absolutely no issue with the way TPS is read on MegaSquirt's. It seems like a solution that has no problem.

Am I wrong here?

My car became a hopeless case when I went below 70 for tpsThresh. That probably reflects something I did wrongly in my install -- perhaps reusing the old wiring for the throttle switch, which was never intended to carry an analog signal. But I don't think it can be just me. There has been a fair bit of interest in this thread -- and not just amongst nerds -- so it seems there may be many pitfalls in getting a really useful TPS signal. Coincidentally, I'm also using a tpsThresh of 40 these days; I would have no difficulty going much lower if I wanted to. What I can say is that the process of accel tuning was much quicker and easier than my previous attempts had been. Things seemed much more consistent. In this case it may just have been me as my experience/understanding grew.

It would be interesting if you were to try the gslender version. Perhaps you'd see no benefit. Then again, maybe the smoothed signal would see the car running even better. This is probably not too attractive when you're already happy with the way the car's running. Maybe try it with your next project ..

Have fun,

Rob.

[gslender]

-- perhaps reusing the old wiring for the throttle switch, which was never intended to carry an analog signal. But I don't think it can be just me. There has been a fair bit of interest in this thread -- and not just amongst nerds -- so it seems there may be many pitfalls in getting a really useful TPS signal.

Yeah, that is interesting as I've spoken to a variety of folks who have battled with trying to get a noise free TPS. Some solve it by replacing the wiring (so clearly having a good ground and quality wire helps) but others, like me, who tried this also found it didn't make any difference - so whilst better wiring helped some folks, it hasn't helped everyone.

I think there is some other factors at play - for me I'm starting to think I have noise on the 12V line that isn't being filtered (that is perhaps introduced by the EGO). My battery is particularly jumpy/noisy and I always thought that was ok, but now I'm not sure... in any case, the improvements I've gained from using software to clean up the inputs has been great!

I just wished I could afford a cheap Osciliscope so that maybe I'd be able to track down where this nosie is coming from....

[racingmini_mtl]

I just wished I could afford a cheap Osciliscope so that maybe I'd be able to track down where this nosie is coming from....

viewtopic.php?f=94&t=38560

[robs]

I’d like to point out again that in theory a 2-pole (done twice) lag filter with the proper “lag %” will provide superior filtering to a simple running average filter.

Thought I'd do another bit of graphing to illustrate what Jason's referring to here -- just so it's clear what he's referring to (or, if I'm wrong, so it's clear I don't have a clue).

Once again using R, I created a series of ramps from 1 to 100 over intervals of 2, 5, 20 and 100 samples. I then ran the result through a Bessel filter, and through a mean of 10 samples at a time. The idea is that the really quick spikes (high frequency) are noise, while the slower ones are signal (along the same lines as Greg's graphic equaliser posting).

bessavg.pdf

Bessel vs Mean signal response

(16.72 KiB) Downloaded 24 times

Looking at the graph you can see that the Bessel filter has at least two advantages. Firstly, it responds much more quickly; secondly it mirrors the slow movements more closely than the mean filter, while still doing a good job of reducing the noise.

This isn't meant to reflect any particular real-world figures. Just thought an illustration might help people understand Jason's posting. In particular, the mean will always lag by 10, because it's only calculated every 10th sample. By taking a different sampling approach it no doubt could be made to look better. Similarly, the choice of 40% for the Bessel lag factor was arbitrary.

Have fun,

Rob.

[robs]

I just wished I could afford a cheap Osciliscope so that maybe I'd be able to track down where this nosie is coming from....

viewtopic.php?f=94&t=38560

Or, for something a bit more up-market, you could go for one of the Rigol scopes: http://www.ebay.com.au/itm/Rigol-Oscillosco-pe-DS1052E-50MHz-2-Chennal-OZ-Seller-/120796616050?pt=BI_Oscilloscopes&hash=item1c200a1572. Mine has been great for the couple of years I've had it. Just recently though text on the display became unreadable (colours went weird), but that's just my luck, everyone else in the world seems happy with theirs.

Have fun,

Rob.

[Peter Florance]

I have this one for field work (the cheaper one was at one time the price of the better one)
http://www.hobbylab.us/
it's a little noisy; the unit that is currently $169.50US is probably a better choice.

The spectrum analyzer alone might be helpful here