> If you have a waveform that is more complicated than a sine wave, IIR filters
> are not the right choice.

This is a bold claim...

> If you have a signal that is only 10,000 points in
> length, an IIR filter is not the right choice.

..and so is this. Could you elaborate, please?

> I think you have performed that
> experiment and shown that it isn't the proper choice.

Do you mean me personally, or DSP'ers in general? If I have,
it is without my own knowledge. Could you point to any excercise
that support such a claim? Preferably one you know I have run
through, though.

> IMHO, the proper choice for problems with a short signal length, say less than
> 16.000.000 points, is the FFT.

OK. And what do you consider to be a long signal?

> Perform a forward FFT, excise the offending
> signal elements, then perform an Inverse FFT.

How many "offending" elements do you expect to find in a
spectrum that contain tens of millions of points? How do you intend
to automize such a process?

> It will probably be necessary
> to taper the ends of the spectrum next to the excised components using a ramp
> or half cosine taper with a few points length, maybe nine, before the IFFT to
> prevent undesirable signal components.

Nine points out of 16 millions; sure, that'll do the trick...
Rune

Reply by John Herman●December 8, 20062006-12-08

If you have a waveform that is more complicated than a sine wave, IIR filters
are not the right choice. If you have a signal that is only 10,000 points in
length, an IIR filter is not the right choice. I think you have performed that
experiment and shown that it isn't the proper choice.
IMHO, the proper choice for problems with a short signal length, say less than
16.000.000 points, is the FFT. Perform a forward FFT, excise the offending
signal elements, then perform an Inverse FFT. It will probably be necessary
to taper the ends of the spectrum next to the excised components using a ramp
or half cosine taper with a few points length, maybe nine, before the IFFT to
prevent undesirable signal components.
In article <1164995848.064811.177320@j72g2000cwa.googlegroups.com>, "Rune
Allnor" <allnor@tele.ntnu.no> wrote:

>
>inna-s skrev:
>> I have a similar problem described by Par in 2005.
>> (Only different frequencies and sample size).
>>
>> I have signal of 10000 length that has undesired frequencies:
>> f=0.0003, 0.0006 (f \in [0 1]) I want to get rid off.
>> I build notch IIR and apply it.
>> To test that it works I first created
>> a synthetic signal (linear sum of harmonics) and apply filter on this
>> data.
>> 1. I know that my IIR design is perfect.
>> 2. I also know that to get zero linear phase I apply filter forw and back
>>
>> sequentially.
>>
>> If my synthetic signal is long enough about 50000, I get very good
>> results
>> in the middle part of the signal from [5000 - 45000]. But I don't deal
>> well
>> at the bounds 5000 samples from both sides.
>> It is clear that since my real signal is 10000, I will not succeed to
>> apply my filter to real data. I should get rid off the boundary effects.
>
>These are very narrow bands, and it is not surprising that
>the end effects last for a few thousand samples.
>
>Why not design a HP filter with stop band 0 - 0.001 and
>passband 0.002 - 0.5 ?
>
>Rune
>

Reply by Rune Allnor●December 1, 20062006-12-01

inna-s skrev:

> I have a similar problem described by Par in 2005.
> (Only different frequencies and sample size).
>
> I have signal of 10000 length that has undesired frequencies:
> f=0.0003, 0.0006 (f \in [0 1]) I want to get rid off.
> I build notch IIR and apply it.
> To test that it works I first created
> a synthetic signal (linear sum of harmonics) and apply filter on this
> data.
> 1. I know that my IIR design is perfect.
> 2. I also know that to get zero linear phase I apply filter forw and back
>
> sequentially.
>
> If my synthetic signal is long enough about 50000, I get very good
> results
> in the middle part of the signal from [5000 - 45000]. But I don't deal
> well
> at the bounds 5000 samples from both sides.
> It is clear that since my real signal is 10000, I will not succeed to
> apply my filter to real data. I should get rid off the boundary effects.

These are very narrow bands, and it is not surprising that
the end effects last for a few thousand samples.
Why not design a HP filter with stop band 0 - 0.001 and
passband 0.002 - 0.5 ?
Rune

Reply by inna-s●December 1, 20062006-12-01

>
>
>inna wrote:
>...
>> I have signal of 10000 length that has undesired frequencies:
>> f=0.003, 0.006 (f \in [0 1]) I want to get rid off.
>> I build notch IIR and apply it.
>> To test that it works I first created
>> a synthetic signal (linear sum of harmonics) and apply filter on this
>> data.
>> 1. I know that my IIR design is perfect.
>> 2. I also know that to get zero linear phase I apply filter forw and

back

>>
>> sequentially.
>>
>> If my synthetic signal is long enough about 50000, I get very good
>> results
>> in the middle part of the signal from [5000 - 45000]. But I don't deal
>> well
>> at the bounds 5000 samples from both sides.
>> It is clear that since my real signal is 10000, I will not succeed to
>> apply my filter to real data. I should get rid off the boundary

effects.

>>
>> Please can you advise me how to resolve these boundary effects.
>> In the old post people advised to use superposition, but I did not
>> catch an idea.
>
>If the sine signal has constant amplitude, phase and frequency, you can
>try to estimate these parameters and then subtract a synthesized sine.
>
>Regards,
>Andor
>
>

Yes, you are right, I used this method as well; it is ok for additive
models, but my real signal is more complicated.
I am more interested now to understand the constraints and pitfalls
of IIR filtering. This method works quite well, but needs currently a
signal of length 50000. I used already filtfilt.m and don't find it
improves something for me.

Reply by inna-s●December 1, 20062006-12-01

>
>
>inna wrote:
>...
>> I have signal of 10000 length that has undesired frequencies:
>> f=0.003, 0.006 (f \in [0 1]) I want to get rid off.
>> I build notch IIR and apply it.
>> To test that it works I first created
>> a synthetic signal (linear sum of harmonics) and apply filter on this
>> data.
>> 1. I know that my IIR design is perfect.
>> 2. I also know that to get zero linear phase I apply filter forw and

back

>>
>> sequentially.
>>
>> If my synthetic signal is long enough about 50000, I get very good
>> results
>> in the middle part of the signal from [5000 - 45000]. But I don't deal
>> well
>> at the bounds 5000 samples from both sides.
>> It is clear that since my real signal is 10000, I will not succeed to
>> apply my filter to real data. I should get rid off the boundary

effects.

>>
>> Please can you advise me how to resolve these boundary effects.
>> In the old post people advised to use superposition, but I did not
>> catch an idea.
>
>If the sine signal has constant amplitude, phase and frequency, you can
>try to estimate these parameters and then subtract a synthesized sine.
>
>Regards,
>Andor
>
>

Yes, you are right, I used this method as well; it is ok for additive
models, but my real signal is more complicated.
I am more interested now to understand the constraints and pitfalls
of IIR filtering. This method works quite well, but needs currently a
signal of length 50000. I used already filtfilt.m and don't find it
improves something for me.

Reply by Andor●December 1, 20062006-12-01

inna wrote:
...

> I have signal of 10000 length that has undesired frequencies:
> f=0.0003, 0.0006 (f \in [0 1]) I want to get rid off.
> I build notch IIR and apply it.
> To test that it works I first created
> a synthetic signal (linear sum of harmonics) and apply filter on this
> data.
> 1. I know that my IIR design is perfect.
> 2. I also know that to get zero linear phase I apply filter forw and back
>
> sequentially.
>
> If my synthetic signal is long enough about 50000, I get very good
> results
> in the middle part of the signal from [5000 - 45000]. But I don't deal
> well
> at the bounds 5000 samples from both sides.
> It is clear that since my real signal is 10000, I will not succeed to
> apply my filter to real data. I should get rid off the boundary effects.
>
> Please can you advise me how to resolve these boundary effects.
> In the old post people advised to use superposition, but I did not
> catch an idea.

If the sine signal has constant amplitude, phase and frequency, you can
try to estimate these parameters and then subtract a synthesized sine.
Regards,
Andor

Reply by inna-s●December 1, 20062006-12-01

I have a similar problem described by Par in 2005.
(Only different frequencies and sample size).
I have signal of 10000 length that has undesired frequencies:
f=0.0003, 0.0006 (f \in [0 1]) I want to get rid off.
I build notch IIR and apply it.
To test that it works I first created
a synthetic signal (linear sum of harmonics) and apply filter on this
data.
1. I know that my IIR design is perfect.
2. I also know that to get zero linear phase I apply filter forw and back
sequentially.
If my synthetic signal is long enough about 50000, I get very good
results
in the middle part of the signal from [5000 - 45000]. But I don't deal
well
at the bounds 5000 samples from both sides.
It is clear that since my real signal is 10000, I will not succeed to
apply my filter to real data. I should get rid off the boundary effects.
Please can you advise me how to resolve these boundary effects.
In the old post people advised to use superposition, but I did not
catch an idea. The reference to filtfilt of matlab was given there
as well and it was claimed that some IEEE paper is mentioned in filtfilt
doc. I checked the Matlab documents, but today there is no any reference
to
a paper. Does anyone know what is this paper? My idea was to make
padding;
repeat the same signal a few times, but I understood from old post that
this is a wrong technique...
I also know from simulation that these boundary effects are related to the
bandwidth of the notch filter. The more narrow filter, the more area is
occupied by the boundary effects. I am curious if there is also some
math. that relates desired frequency, bandwidth etc. and the boundary
effects zone.
Also if this problem has a standard name please can you correct me,
as I will try to make search using a right terminology.
Thanks in advance,
Inna.