Thermocouple bricklet readings "sticking"

[zef]

Hi all,

I've been working on a PID temperature controller and have been loving the Tinkerforge system overall.

The only issue I haven't been able to solve is an intermittent "sticking" of readings from my thermocouple bricklet V2 with a K-type probe. I'm using the thermocouple callback to give me a value every 1 second as reccomended by the docs, with 'value_has_to_change' set to False.

This works great 99% of the time. But occassionally and apparently randomly, the bricklet returns an identical reading for a short period of time. You can see in my attached image: the two little plateaus in the graph, even though the real temperature is steadily rising (the X-axis here is 1 second per point). In this example I was using 16-sample averaging, but the issue seems to persist for all averaging values. The data on the graph is processed obviously, but I've confirmed that the thermocouple callback itself is being called with identical values.

I haven't been able to pin down why this is happening. I'm using the Python bindings on a Hat Zero brick alongside a SSR and LCD bricklet. The only thing that seems to help is reducing the frequency of the callback, but I suspect it's just masking the problem. I'm fairly sure the probe and connections are fine: I don't experience the sticking with an external thermocouple reader, and the apparatus isn't moving around or subject to any other external interferrence that I can see.

Has anyone else experienced anything like this? Anything I could try? My next idea would be to try an isolator bricklet, but that would just be a guess! Other ideas: possible performance issues on my Pi Zero, causing the various callbacks in my script to compete with each other? Not sure. From what I can see the callback is firing on time, just with the same value over and over!

The thermocouple configuration section of my code is here, for reference: https://github.com/BenVosper/heated/blob/master/regulated.py#L238:L256

Let me know if you need any more info about my setup. Thanks!

[rtrbt]

Hi,

Do you see any activity on the error state callback while the value is stuck? The firmware does not update the temperature value while the open circuit error is set.

[zef]

@rtrbt Thanks for pointing that out, I should have seen that in the docs!

It does seem that's what's happening. I can see the open-circuit errors happening in sync with the little plateaus in the readings. That makes sense.

But what could be causing the open-circuit errors? Some interference in the system? I can see it happening with two different probes. Admittedly they're not particularly expensive or high quality probes, but I don't have any reason to suspect they're both faulty.

Nothing is moving when I see errors, so I'm not sure it's any physical connection at fault either.

Could it be some quirk of my wiring / enclosure? I've got 240VAC coming into the enclosure and being converted to 5v for the brick and bricklets via a high quality DC power supply with plenty of headroom, but could it be too electrically noisy within the enclosure?

Does that sound likely at all? I wonder if an isolator bricklet would help

[rtrbt]

18 hours ago, zef said:

Does that sound likely at all?

Maybe. As a simple test, you could build a minimal setup with only the bricklet and the probe (attached to the HAT and Pi or, if you have one, a Master Brick) without an enclosure, connect to the Brick or Pi with Brick Viewer and see if the bricklet gets stuck again. Of course this depends on how often the errors happen.

[zef]

I've confirmed that running the brick and bricklets with an external 5V supply seemed to prevent the error states. So possibly some interferrence caused by the DC power supply within the enclosure.

I tried some rudimentary shielding around the thermocouple bricklet, but that didn't seem to help on its own.

So I think I'll try out an isolator bricklet to see if that will improve the stability. Worst case I'll just have to find a different power supply or change my enclosure around a little.

Thanks for pointing me in the right direction!

EDIT: Will the isolator bricklet be comptible with my HAT Zero if I connect it with a 7p-7p cable? The docs only mention it working with a 7p-10p to the brick

bearbeitet May 5, 2021 at 22:11 von zef

[rtrbt]

The Isolator Bricklet is compatible with the HAT Zero. The docs are a bit outdated in this case, as they often assume, that all Bricks come with the 10p connector.

[zef]

^ Thanks for clarifying.

I gave the isolator bricklet a go today. Unfortunately it didn't seemt to prevent the error states. I was getting the same intermittent dropouts with a variety of different probes / power supplies.

Maybe I'll try a slightly better probe. That or perhaps reshuffle things inside the enclosure a bit

[zef]

Had a breakthrough. Nothing wrong with the probes / setup. The probes I'm using have stainless-steel cladding on their leads, and for whatever reason connecting the cladding to DC ground totally eliminated the error states. This is without the isolator brick in the system.

I'm not entirely sure why this fixed it, but I haven't had a single error reported with the new grounding in place. I guess some kind of interference / ground loop in the cladding screwing up the thermocouple voltages?

[seismicvibrations]

Hello zef,

I have also seen similar issues with the thermocouple bricklets I control for a setup and I suspect the cause might be similar. Would you mind posting a photo of your probe and what you mean by "cladding"? I have long suspected grounding problems but when I look at my probes I don't see what could be grounded.

Thanks in advance.

[zef]

@seismicvibrations Maybe "cladding" isn't the right word! The high-temperature probes I'm using came with stainless-steel braid on the outside of their leads, over the top of the ceramic / glass fibre insulation of the wires themselves. This is the one I'm currently using:



I found that connecting this outer steel braiding to DC ground greatly reduced the error states. I did this simply with a crocodile clip clamping onto the lead at the connector end, close to the connector itself.

I also have probes like this which have a steel cover over the probe tip itself, also with the steel braiding over the lead:


Grounding the braid seemed to have the same effect for me on these. The steel tip cover is electrically connected to the braiding on the one I have, so perhaps if your probes don't have the braiding, you could try grounding the tip itself?

[seismicvibrations]

Thanks @zef! Now I really understand what you meant. I am not sure my probes are constructed the same but I am looking into it, and will see what I can connect to ground.

[zef]

Hi all,

My little digital oven has been working great for many years! I've regularly been using it to reach temperatures of over 800°C with no major issues. I still get very intermittent "open circuit" errors reported by the thermocouple bricklet, but these only last for a few seconds before everything returns to normal, which doesn't affect the operation of the oven.

But I've recently needed to use temperatures of up to 1100°C for certain applications. In this region above 800°C, the open circuit dropouts become much more frequent and seem to occur more and more as the temperature increases. As I approach 1100°C the bricklet reports open circuit the majority of the time only occasional readings coming through, which isn't enough for reliable temperature control. The bricklet reports open circuit even if no power is being sent to the heating element, so I don't think it's an issue with AC interference on the probe side.

I don't believe there's a problem with the physical connections. I can read the thermocouple using an external reader and I don't experience it going open circuit: it seems to work fine at the higher temperatures. Which leads me to believe there's some kind of electrical or software issue. Can anyone think of anything that would cause the thermocouple bricklet to read open circuit erroneously?

I'm using an isolator bricklet in between the thermocouple bricklet and the HAT Zero brick.

Things I've tried with no success:

•  Various combinations of grounding / ungrounding the thermocouple cladding, device enclosure etc.
•  Physically rearranging device layout to separate the thermocouple cables and bricklets from possible AC interferrence
•  Updating Tinkerforge Python bindings and OS to latest versions

Things I haven't yet tried:

• Use a better thermocouple probe - I've ordered a high quality probe from Omega, just to rule out my current cheap probe
• Powering the Pi Zero using a completely separate DC power source (maybe eliminating AC interference inside the enclosure?)
• Removing the isolator bricklet

 

Can anyone think of anything else I could try? The temperature-dependent effect is very strange! Let me know if I can provide any more info on my setup. Thanks!

[zef]

An update.

I've reconfigured the internal layout of my enclosure to minimise AC interference with the thermocouple bricklet, and replaced the thermocouple itself with a high quality Omega XC probe. This has reduced the frequency of the open-circuit readings at temperatures from around 800°C to 1000°C. So that's good!

But I'm still getting intermittent error state readings at temperatures over 1000°C. So I must be approaching some fundamental limit of my setup.

I notice that when an error state is reported, this seems to stay active for a relatively constant period of 5 seconds. Maybe someone more familiar with the thermocouple bricklet firmware can answer the following questions for me. I've had a quick look at the firmware code itself but it's pretty complex!

1. After an error state is reached, how long does the firmware wait before checking again? ie. Is there a minimum time between calls to the CALLBACK_ERROR_STATE callback?
2. Does the frequency of the CALLBACK_TEMPERATURE callback affect how quickly the error state might be cleared? I currently have the temperature read callback configured with a 1 second period, so I'm usually getting 4 or 5 "stuck" temperature readings when an error state is triggered.

I'm just wondering if this 5 second effect I'm seeing might be caused by the firmware, or something I might be doing in my code.

[borg]

I did some digging in the source code of the Bricklet and the datasheet of the MAX31856: As far as i can tell the status register is read in a loop and the reading of new temperatures is stopped when there is an error in the status register and started again when the error is gone. So i think the 5 seconds are internally in the MAX31856 (i can not find any data on this in the datasheet though).

But i found something else. The chip supports different kinds of open circuit detection:

The Bricklet currently always configures this to 01, which has the smallest test time.

I made you two firmwares for testing: One has OCFAULT set to 11, which should be more forgiving. The other firmware has the Open-Circuit detection disabled (OCFAULT set to 00). Since you are sure that the open circuit detection is a false positive in your case, maybe it is the easiest to just disable it.

If any of the settings helps for you, i can add this setting as a new configuration setter to the API.

thermocouple-v2-bricklet-firmware-ocfault-00.zbin thermocouple-v2-bricklet-firmware-ocfault-11.zbin