ehFeedback error

llglick's Avatar

llglick

09 Jul, 2021 08:16 PM

Hi Chris,
We've been sporadically getting the attached error (ehFeedback error, see stdout), and shortly after mWorks crashes.
Do you have thoughts on what might be causing it? Anything we can do other than restarting the computer/labjack?
Thanks,
Lindsey.

  1. Support Staff 1 Posted by Christopher Sta... on 09 Jul, 2021 08:30 PM

    Christopher Stawarz's Avatar

    Hi Lindsey,

    (Sorry for creating multiple discussions. The support site was being flaky, but it seems better now.)

    I can't say I have much insight in to the problem. It seems like a simple write to a digital output shouldn't be failing, and certainly such a failure shouldn't lead to MWServer crashing. This makes me think there's memory corruption happening somewhere, and the write failure and crash are side effects of that.

    Are you sure you've recompiled your plugin against the version of MWorks you're using? If not, that's definitely a potential source of memory corruption.

    You might try following the suggestion in the error message and looking at stdout. The simplest way to do this is to launch MWServer from the command line via Terminal:

    $ /Applications/MWServer.app/Contents/MacOS/MWServer
    

    Any messages written to standard output or standard error will appear in the Terminal window. I don't know if you'll see anything helpful, but it's possible you'll get more detail about the write failure.

    Cheers,
    Chris

  2. 2 Posted by llglick on 13 Jul, 2021 04:09 PM

    llglick's Avatar

    Hi Chris,
    Here's the output from stdout.
    Do you think it's a labjack failure- should we reach out to them?
    Thanks,
    Lindsey.

  3. Support Staff 3 Posted by Christopher Sta... on 13 Jul, 2021 06:29 PM

    Christopher Stawarz's Avatar

    Hi Lindsey,

    The image you sent is of MWServer's console window. (The error message in it indicates that you have more than one instance of MWServer running.) The standard output will be in the Terminal window from which you launched MWServer.

    To see MWServer's standard output:

    1. Close all running instances of MWServer.
    2. Open the Terminal application (in /Applications/Utilities on your Mac).
    3. In the Terminal window, enter

      /Applications/MWServer.app/Contents/MacOS/MWServer
      

      and press return. MWServer should launch.

    4. Run your experiment. MWServer's standard output (if there is any) will appear in the Terminal window.

    Chris

  4. 4 Posted by llglick on 14 Jul, 2021 04:09 PM

    llglick's Avatar

    Hi Chris,
    Thanks for clarifying. Here's the stdout from the Terminal.
    Interestingly (frustratingly) it was a different computer this time.
    Lindsey.

  5. Support Staff 5 Posted by Christopher Sta... on 14 Jul, 2021 08:25 PM

    Christopher Stawarz's Avatar

    Hi Lindsey,

    Thanks for sending that. Do you know if the "response buffer is too small" message happens at the same time as the "bug: writing digital output low" message in the MWServer console?

    In any case, the "response buffer is too small" message indicates failure to receive a response from the U6, which is worrisome. Have you changed anything in your setup recently (e.g. macOS version, MWorks version, other USB devices)? Are you seeing the "bug: writing digital output low" with multiple U6 devices or just one?

    It might be worthwhile to contact LabJack and see if they have any insight or recommendations.

    Chris

  6. 6 Posted by llglick on 15 Jul, 2021 03:37 PM

    llglick's Avatar

    Hi Chris,
    We haven't caught the moment that these errors appear (often they're ~20
    minutes into an experiment) so I don't know about the relative timing, but
    I assume they're close.
    Right now, this is happening on two different Labjacks. We have not
    changed OS, Mworks or anything lately. We did swap out a computer
    yesterday (with the same labjack) and it still crashed. There is some
    (anecdotal) link to a specific variable condition, but I've been digging
    through the code and can't figure out why it would matter.
    We're getting in touch with Labjack now.
    Thanks,
    Lindsey.

  7. 7 Posted by llglick on 16 Jul, 2021 01:44 PM

    llglick's Avatar

    Hi Chris,
    Labjack says: "The error indicates that a transfer was bigger than the
    maximum transfer size (64 bytes for our USB endpoints). You should ensure
    that your transfers are always sized to be a multiple of the maximum packet
    size. Also be sure you always send an even number of bytes (appending a
    byte of 0x00 if the packet has an odd number of bytes)."
    Any thoughts on what might cause transfers to be larger than expected?
    Let me know if it'd be helpful to see our experiment code.
    Lindsey.

  8. 8 Posted by mhisted on 16 Jul, 2021 03:31 PM

    mhisted's Avatar

    Hi Chris and Lindsey,

    We have been seeing errors like this too.

    The relevant code here is the code I wrote a long time ago to control the labjack, so it will be shared in Lindsey’s current plug-in and ours. So hopefully we can find the same fix.

    We have one machine whose hardware we think is failing, and are replacing it. I don’t know yet whether this is happening just on that rig or on others, and I’m asking my team now.

  9. Support Staff 9 Posted by Christopher Sta... on 19 Jul, 2021 02:18 PM

    Christopher Stawarz's Avatar

    Hi Lindsey & Mark,

    Based on your experiences, my current suspicion is that this issue stems from the ancient version of libusb that LabJack includes in their Exodriver package for macOS. (I just reinstalled the current release, and the bundled libusb binary has a creation date in 2012.)

    Would either of you be willing to try a version of your U6 plugin built against a newer libusb? Since version 0.10, MWorks has shipped with a libusb binary. (It's needed for the DATAPixx interface.) If you're running MWorks 0.10 or later, then I can just update your plugin code to build against MWorks' bundled libusb, which is very recent. If you're using an older version of MWorks, things will be slightly more complicated, but we can make it work.

    Thanks,
    Chris

  10. 10 Posted by llglick on 19 Jul, 2021 02:44 PM

    llglick's Avatar

    Hi Chris,
    We are currently using 0.10 and would be happy to try an updated plugin.
    I think you already have access to our plugin repository? It has diverged
    a bit from Mark's, so you may have to make the changes separately...
    Lindsey.

  11. Support Staff 11 Posted by Christopher Sta... on 19 Jul, 2021 04:05 PM

    Christopher Stawarz's Avatar

    Hi Lindsey,

    We are currently using 0.10 and would be happy to try an updated plugin.

    OK, great. I've attached a build of your plugin against MWorks 0.10 and its bundled libusb. To install the plugin, unpack the ZIP file, copy LabJackU6Plugin.bundle to the folder /Library/Application Support/MWorks/Plugins/Core, and restart MWServer. Once that's done, please try running your experiment, and let us know how things go.

    Thanks,
    Chris

  12. 12 Posted by llglick on 19 Jul, 2021 07:40 PM

    llglick's Avatar

    Hi Chris,
    Tried the new plugin but it still crashed.
    Lindsey.

  13. Support Staff 13 Posted by Christopher Sta... on 19 Jul, 2021 08:05 PM

    Christopher Stawarz's Avatar

    Hi Lindsey,

    Tried the new plugin but it still crashed.

    OK, thanks for trying. I guess that leaves the plugin itself as the most likely source of the problem.

    Let me know if it'd be helpful to see our experiment code.

    I think that would be very helpful at this point. Can you send me the experiment that triggers the crash?

    If possible, please include all associated files (images, sounds, etc.), so that I can reproduce the runtime environment as closely as possible. Also, I see that your plugin reads calibration info from some hard-coded paths (e.g. /Users/hullglick/Documents/Calibration_Table/led.txt.) Can you also include a copy of whichever file you're using for the example experiment?

    Thanks,
    Chris

  14. 14 Posted by llglick on 19 Jul, 2021 08:28 PM

    llglick's Avatar

    Great- thanks.
    Here's the bundle. I think it has all of the (non-matlab) dependent files,
    but please let me know if it's missing anything.
    I've also included an example variable set- i485.xml. If you set
    "doTestRobot" to 1, it will run on its own. Also- we've found that setting
    "doFeedbackMotion" to 0 increases the likelihood of crashing (which is
    strange because this should be less overhead)- so hopefully that will help
    you replicate.
    Lindsey.

  15. Support Staff 15 Posted by Christopher Sta... on 19 Jul, 2021 08:55 PM

    Christopher Stawarz's Avatar

    Thanks, I'm able to run the experiment. I haven't seen any errors yet, but I'll keep it running and will hopefully get a crash eventually.

    Chris

  16. 16 Posted by mhisted on 19 Jul, 2021 09:28 PM

    mhisted's Avatar

    Hi Chris and Lindsey,

    I suspect that this could be a hardware issue – with the Labjack, or the USB hub, or perhaps the Mac. If Chris can't get the crash replicated, perhaps he should look to remote in.
    We replaced one of our Macs, which we think eliminated one source of these issues on one rig, and we're watching to see if we see this again on the second rig that was suspect.

    Mark

  17. 17 Posted by llglick on 19 Jul, 2021 09:36 PM

    llglick's Avatar

    Hi Mark,
    We replaced the Mac and it didn't help, but it could easily be the Labjack
    or hub. I've got new labjacks arriving on Weds, and I think I have a spare
    hub that I can try tomorrow.
    Thanks,
    Lindsey.

  18. Support Staff 18 Posted by Christopher Sta... on 20 Jul, 2021 02:58 PM

    Christopher Stawarz's Avatar

    Quick update: I have now seen the ehFeedback error. MWServer didn't subsequently crash, but it did seem to deadlock. I'm continuing to investigate.

    Chris

  19. 19 Posted by llglick on 20 Jul, 2021 04:47 PM

    llglick's Avatar

    Ok- good to know.
    Now that you've at least partially replicated, I'm leaning away from a
    hardware issue (though could cause increased susceptibility). But just to
    follow up on our end- we eliminated the USB hub (just plugged the labjack
    directly into the mac) and it still crashed.
    Lindsey.

  20. Support Staff 20 Posted by Christopher Sta... on 21 Jul, 2021 01:16 PM

    Christopher Stawarz's Avatar

    Hi Lindsey,

    I think I found the problem:

    In the method LabJackU6Device::stopDeviceIO, there are two calls to ljU6WriteDO that execute without holding ljU6DriverLock. This allows them to overlap with other commands sent to the LabJack device in update_lever, which runs on a separate thread associated with pollScheduleNode. Since stopDeviceIO is called at the end of every trial, an overlap eventually happens. This causes commands to the device to be interleaved, leading to unexpected responses and the associated errors.

    The fix is simply to acquire the lock before calling ljU6WriteDO. Prior to making the fix, I found that the error consistently occurred 10-15 minutes after starting the experiment. Since making the fix, I've run the experiment continuously for up to 1.5 hours, and I haven't see any errors.

    I've attached another MWorks 0.10 build of the plugin that includes the fix. Can you try this out and see if it resolves the issue for you, too? If so, I'll submit a pull request with the changes on GitHub.

    Mark: The problematic code in Lindsey's plugin isn't present in yours, so clearly this isn't the cause of the issues you're seeing. It's possible that there's a similar problem elsewhere, or it could be the case that your issues are hardware-related, as you've suggested.

    Chris

  21. 21 Posted by llglick on 21 Jul, 2021 08:55 PM

    llglick's Avatar

    Hi Chris,
    So, good news is that MWorks no longer crashes with the new Labjack plugin.
    However, now there seems to be a problem with the communication with
    Matlab. I'm still not sure what the exact issue is, but our plotting
    function gets hung up on variables it should recognize. I'm fairly positive
    that it's related to the new plugin (after this happened on the first
    computer I updated, I tested the old plugin on a different computer and
    then installed the new plugin and tested again, and it no longer worked).
    Maybe related- the python script that we use to load our variable set also
    no longer works (this issue came up after the first plugin you sent us, but
    I didn't make the connection until now).
    There are few Matlab files that you would need to replicate, but I'm happy
    to send if it's helpful.
    Thanks,
    Lindsey.

  22. Support Staff 22 Posted by Christopher Sta... on 22 Jul, 2021 01:07 PM

    Christopher Stawarz's Avatar

    Hi Lindsey,

    I just realized I made a mistake with both of the previous plugin builds I sent you: I had changed the path to the file led.txt, and I built the plugin for you without changing it back to its original value. I've attached a new build of the plugin with the correct path restored.

    Is it possible that the bad path was the source of your MATLAB and Python issues?

    Chris

  23. 23 Posted by llglick on 22 Jul, 2021 01:50 PM

    llglick's Avatar

    Hi Chris,
    Turns out the Matlab bug was on our side after all (the python bug is still
    a mystery). But I got your new plug-in with the updated path and everything
    looks great!
    Thank you so much for your help.
    Lindsey.

  24. Support Staff 24 Posted by Christopher Sta... on 29 Jul, 2021 02:19 PM

    Christopher Stawarz's Avatar

    Hi Lindsey,

    I've submitted a pull request with the bug fix. Also in that PR are changes that allow the plugin to natively support both Intel and Apple Silicon processors, as we've discussed elsewhere.

    Cheers,
    Chris

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Already uploaded files

  • Image_from_iOS.jpg 1.42 MB

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac