ehFeedback error
Hi Chris,
We've been sporadically getting the attached error (ehFeedback error, see stdout), and shortly after mWorks crashes.
Do you have thoughts on what might be causing it? Anything we can do other than restarting the computer/labjack?
Thanks,
Lindsey.
-
Image_from_iOS.jpg
1.42 MB
Image_from_iOS.jpg
Keyboard shortcuts
Generic
| ? | Show this help |
|---|---|
| ESC | Blurs the current field |
Comment Form
| r | Focus the comment reply box |
|---|---|
| ^ + ↩ | Submit the comment |
You can use Command ⌘ instead of Control ^ on Mac
Support Staff 1 Posted by Christopher Sta... on 09 Jul, 2021 08:30 PM
Hi Lindsey,
(Sorry for creating multiple discussions. The support site was being flaky, but it seems better now.)
I can't say I have much insight in to the problem. It seems like a simple write to a digital output shouldn't be failing, and certainly such a failure shouldn't lead to MWServer crashing. This makes me think there's memory corruption happening somewhere, and the write failure and crash are side effects of that.
Are you sure you've recompiled your plugin against the version of MWorks you're using? If not, that's definitely a potential source of memory corruption.
You might try following the suggestion in the error message and looking at stdout. The simplest way to do this is to launch MWServer from the command line via Terminal:
Any messages written to standard output or standard error will appear in the Terminal window. I don't know if you'll see anything helpful, but it's possible you'll get more detail about the write failure.
Cheers,
Chris
2 Posted by llglick on 13 Jul, 2021 04:09 PM
Hi Chris,
Here's the output from stdout.
Do you think it's a labjack failure- should we reach out to them?
Thanks,
Lindsey.
Support Staff 3 Posted by Christopher Sta... on 13 Jul, 2021 06:29 PM
Hi Lindsey,
The image you sent is of MWServer's console window. (The error message in it indicates that you have more than one instance of MWServer running.) The standard output will be in the Terminal window from which you launched MWServer.
To see MWServer's standard output:
/Applications/Utilitieson your Mac).In the Terminal window, enter
and press return. MWServer should launch.
Run your experiment. MWServer's standard output (if there is any) will appear in the Terminal window.
Chris
4 Posted by llglick on 14 Jul, 2021 04:09 PM
Hi Chris,
Thanks for clarifying. Here's the stdout from the Terminal.
Interestingly (frustratingly) it was a different computer this time.
Lindsey.
Support Staff 5 Posted by Christopher Sta... on 14 Jul, 2021 08:25 PM
Hi Lindsey,
Thanks for sending that. Do you know if the "response buffer is too small" message happens at the same time as the "bug: writing digital output low" message in the MWServer console?
In any case, the "response buffer is too small" message indicates failure to receive a response from the U6, which is worrisome. Have you changed anything in your setup recently (e.g. macOS version, MWorks version, other USB devices)? Are you seeing the "bug: writing digital output low" with multiple U6 devices or just one?
It might be worthwhile to contact LabJack and see if they have any insight or recommendations.
Chris
6 Posted by llglick on 15 Jul, 2021 03:37 PM
Hi Chris,
We haven't caught the moment that these errors appear (often they're ~20
minutes into an experiment) so I don't know about the relative timing, but
I assume they're close.
Right now, this is happening on two different Labjacks. We have not
changed OS, Mworks or anything lately. We did swap out a computer
yesterday (with the same labjack) and it still crashed. There is some
(anecdotal) link to a specific variable condition, but I've been digging
through the code and can't figure out why it would matter.
We're getting in touch with Labjack now.
Thanks,
Lindsey.
7 Posted by llglick on 16 Jul, 2021 01:44 PM
Hi Chris,
Labjack says: "The error indicates that a transfer was bigger than the
maximum transfer size (64 bytes for our USB endpoints). You should ensure
that your transfers are always sized to be a multiple of the maximum packet
size. Also be sure you always send an even number of bytes (appending a
byte of 0x00 if the packet has an odd number of bytes)."
Any thoughts on what might cause transfers to be larger than expected?
Let me know if it'd be helpful to see our experiment code.
Lindsey.
8 Posted by mhisted on 16 Jul, 2021 03:31 PM
Hi Chris and Lindsey,
We have been seeing errors like this too.
The relevant code here is the code I wrote a long time ago to control the labjack, so it will be shared in Lindsey’s current plug-in and ours. So hopefully we can find the same fix.
We have one machine whose hardware we think is failing, and are replacing it. I don’t know yet whether this is happening just on that rig or on others, and I’m asking my team now.
Support Staff 9 Posted by Christopher Sta... on 19 Jul, 2021 02:18 PM
Hi Lindsey & Mark,
Based on your experiences, my current suspicion is that this issue stems from the ancient version of libusb that LabJack includes in their Exodriver package for macOS. (I just reinstalled the current release, and the bundled libusb binary has a creation date in 2012.)
Would either of you be willing to try a version of your U6 plugin built against a newer libusb? Since version 0.10, MWorks has shipped with a libusb binary. (It's needed for the DATAPixx interface.) If you're running MWorks 0.10 or later, then I can just update your plugin code to build against MWorks' bundled libusb, which is very recent. If you're using an older version of MWorks, things will be slightly more complicated, but we can make it work.
Thanks,
Chris
10 Posted by llglick on 19 Jul, 2021 02:44 PM
Hi Chris,
We are currently using 0.10 and would be happy to try an updated plugin.
I think you already have access to our plugin repository? It has diverged
a bit from Mark's, so you may have to make the changes separately...
Lindsey.
Support Staff 11 Posted by Christopher Sta... on 19 Jul, 2021 04:05 PM
Hi Lindsey,
OK, great. I've attached a build of your plugin against MWorks 0.10 and its bundled libusb. To install the plugin, unpack the ZIP file, copy LabJackU6Plugin.bundle to the folder
/Library/Application Support/MWorks/Plugins/Core, and restart MWServer. Once that's done, please try running your experiment, and let us know how things go.Thanks,
Chris
12 Posted by llglick on 19 Jul, 2021 07:40 PM
Hi Chris,
Tried the new plugin but it still crashed.
Lindsey.
Support Staff 13 Posted by Christopher Sta... on 19 Jul, 2021 08:05 PM
Hi Lindsey,
OK, thanks for trying. I guess that leaves the plugin itself as the most likely source of the problem.
I think that would be very helpful at this point. Can you send me the experiment that triggers the crash?
If possible, please include all associated files (images, sounds, etc.), so that I can reproduce the runtime environment as closely as possible. Also, I see that your plugin reads calibration info from some hard-coded paths (e.g.
/Users/hullglick/Documents/Calibration_Table/led.txt.) Can you also include a copy of whichever file you're using for the example experiment?Thanks,
Chris
14 Posted by llglick on 19 Jul, 2021 08:28 PM
Great- thanks.
Here's the bundle. I think it has all of the (non-matlab) dependent files,
but please let me know if it's missing anything.
I've also included an example variable set- i485.xml. If you set
"doTestRobot" to 1, it will run on its own. Also- we've found that setting
"doFeedbackMotion" to 0 increases the likelihood of crashing (which is
strange because this should be less overhead)- so hopefully that will help
you replicate.
Lindsey.
Support Staff 15 Posted by Christopher Sta... on 19 Jul, 2021 08:55 PM
Thanks, I'm able to run the experiment. I haven't seen any errors yet, but I'll keep it running and will hopefully get a crash eventually.
Chris
16 Posted by mhisted on 19 Jul, 2021 09:28 PM
Hi Chris and Lindsey,
I suspect that this could be a hardware issue – with the Labjack, or the USB hub, or perhaps the Mac. If Chris can't get the crash replicated, perhaps he should look to remote in.
We replaced one of our Macs, which we think eliminated one source of these issues on one rig, and we're watching to see if we see this again on the second rig that was suspect.
Mark
17 Posted by llglick on 19 Jul, 2021 09:36 PM
Hi Mark,
We replaced the Mac and it didn't help, but it could easily be the Labjack
or hub. I've got new labjacks arriving on Weds, and I think I have a spare
hub that I can try tomorrow.
Thanks,
Lindsey.
Support Staff 18 Posted by Christopher Sta... on 20 Jul, 2021 02:58 PM
Quick update: I have now seen the ehFeedback error. MWServer didn't subsequently crash, but it did seem to deadlock. I'm continuing to investigate.
Chris
19 Posted by llglick on 20 Jul, 2021 04:47 PM
Ok- good to know.
Now that you've at least partially replicated, I'm leaning away from a
hardware issue (though could cause increased susceptibility). But just to
follow up on our end- we eliminated the USB hub (just plugged the labjack
directly into the mac) and it still crashed.
Lindsey.
Support Staff 20 Posted by Christopher Sta... on 21 Jul, 2021 01:16 PM
Hi Lindsey,
I think I found the problem:
In the method
LabJackU6Device::stopDeviceIO, there are two calls toljU6WriteDOthat execute without holdingljU6DriverLock. This allows them to overlap with other commands sent to the LabJack device inupdate_lever, which runs on a separate thread associated withpollScheduleNode. SincestopDeviceIOis called at the end of every trial, an overlap eventually happens. This causes commands to the device to be interleaved, leading to unexpected responses and the associated errors.The fix is simply to acquire the lock before calling
ljU6WriteDO. Prior to making the fix, I found that the error consistently occurred 10-15 minutes after starting the experiment. Since making the fix, I've run the experiment continuously for up to 1.5 hours, and I haven't see any errors.I've attached another MWorks 0.10 build of the plugin that includes the fix. Can you try this out and see if it resolves the issue for you, too? If so, I'll submit a pull request with the changes on GitHub.
Mark: The problematic code in Lindsey's plugin isn't present in yours, so clearly this isn't the cause of the issues you're seeing. It's possible that there's a similar problem elsewhere, or it could be the case that your issues are hardware-related, as you've suggested.
Chris
21 Posted by llglick on 21 Jul, 2021 08:55 PM
Hi Chris,
So, good news is that MWorks no longer crashes with the new Labjack plugin.
However, now there seems to be a problem with the communication with
Matlab. I'm still not sure what the exact issue is, but our plotting
function gets hung up on variables it should recognize. I'm fairly positive
that it's related to the new plugin (after this happened on the first
computer I updated, I tested the old plugin on a different computer and
then installed the new plugin and tested again, and it no longer worked).
Maybe related- the python script that we use to load our variable set also
no longer works (this issue came up after the first plugin you sent us, but
I didn't make the connection until now).
There are few Matlab files that you would need to replicate, but I'm happy
to send if it's helpful.
Thanks,
Lindsey.
Support Staff 22 Posted by Christopher Sta... on 22 Jul, 2021 01:07 PM
Hi Lindsey,
I just realized I made a mistake with both of the previous plugin builds I sent you: I had changed the path to the file
led.txt, and I built the plugin for you without changing it back to its original value. I've attached a new build of the plugin with the correct path restored.Is it possible that the bad path was the source of your MATLAB and Python issues?
Chris
23 Posted by llglick on 22 Jul, 2021 01:50 PM
Hi Chris,
Turns out the Matlab bug was on our side after all (the python bug is still
a mystery). But I got your new plug-in with the updated path and everything
looks great!
Thank you so much for your help.
Lindsey.
Support Staff 24 Posted by Christopher Sta... on 29 Jul, 2021 02:19 PM
Hi Lindsey,
I've submitted a pull request with the bug fix. Also in that PR are changes that allow the plugin to natively support both Intel and Apple Silicon processors, as we've discussed elsewhere.
Cheers,
Chris