NiDAQ error on new Mac Pro

ereming's Avatar

ereming

06 Jan, 2015 02:51 PM

Hi Chris,

We set up a new rig with a Mac Pro trash can running 10.9.5 (downgraded) with a NiDAQ 6251 PCIe card in a thunderbolt conversion housing. I'm getting the following error that I'm not seeing in our other rig:

00:13:58: ERROR: NIDAQ error: Specified operation did not complete, because the specified timeout expired. (-200474)

00:13:58: WARNING: Scheduled task (/Users/mwdev/Documents/mworks_buildbot/slave/build_all/build/plugins/core/NIDAQ/NIDAQPlugin/NIDAQDevice.cpp:372) falling behind, dropping 3 scheduled executions (priority = 95, interval = 3000, task = 0x6000001c5dc0)

Do you have any idea of what could be causing this? Our other rigs are running 10.8, so I'm thinking the issue could be either Mavericks or Thunderbolt.

Thanks,
Evan

  1. Support Staff 1 Posted by Christopher Sta... on 06 Jan, 2015 03:11 PM

    Christopher Stawarz's Avatar

    Hi Evan,

    My first guess is that it's a Thunderbolt issue. Have you confirmed with someone at NI that your DAQ is supposed to work over Thunderbolt? (I've investigated it in the past but never found a definitive answer.)

    Also, is this an intermittent error, or are all I/O operations failing? What specific I/O type(s) have you tried (i.e. analog/digital, input/output)?

    Thanks,
    Chris

  2. 2 Posted by ereming on 07 Jan, 2015 05:16 PM

    ereming's Avatar

    Hi Chris,

    I spoke with an NI sales guy who said the adapter should work, so I guess I'll open up a service request with NI to see if they can figure out what the issue is. Are these errors enough for them to work with or do you have a more detailed explanation for what the problem is for MWorks?

    Thanks,
    Evan

  3. Support Staff 3 Posted by Christopher Sta... on 07 Jan, 2015 07:59 PM

    Christopher Stawarz's Avatar

    Hi Evan,

    It looks like some read or write operation is timing out. More specifically, one of the following functions is returning error code DAQmxErrorOperationTimedOut (-200474):

    DAQmxBaseReadAnalogF64
    DAQmxBaseWriteAnalogF64
    DAQmxBaseReadDigitalU32
    DAQmxBaseWriteDigitalU32
    DAQmxBaseReadCounterScalarU32
    

    If all you're doing is writing a digital output, then the culprit must be DAQmxBaseWriteDigitalU32.

    That's probably the most I can say without running your experiment in the debugger and observing the error there.

    Cheers,
    Chris

  4. 4 Posted by ereming on 20 Jan, 2015 06:59 PM

    ereming's Avatar

    Hi Chris,

    I've spoken with Apple, NI, and mLOGIC (the PCIe to Thunderbolt expansion chassis manufacturer) about the issue. After having me run several tests, the NI guy thinks the problem is with the converter. An engineer at mLOGIC pointed me to this document from Apple:

    https://developer.apple.com/library/mac/documentation/HardwareDrive...

    Which mentions that PCI drivers may need to be modified to account for the latency added by the thunderbolt connection. There could be additional latency from the converter, so the NI drivers might need to be modified. I was wondering though, are there any timers on the MWorks side that could be modified?

    Thanks,
    Evan

  5. Support Staff 5 Posted by Christopher Sta... on 21 Jan, 2015 02:59 PM

    Christopher Stawarz's Avatar

    Hi Evan,

    I was wondering though, are there any timers on the MWorks side that could be modified?

    Yes. MWorks specifies the timeout for the read/write functions. Currently, it's set to update_interval (the reasoning being that the current operation must complete before we need to start the next one), but we can change it to whatever we want. Do you want me to make the timeout value a parameter that can be set by your experiment?

    Chris

  6. 6 Posted by ereming on 21 Jan, 2015 03:16 PM

    ereming's Avatar

    Oh, in that case no. I was thinking we could extend the MWorks timeout if it was on the order of microseconds.

    Evan

  7. 7 Posted by ereming on 28 Jan, 2015 05:05 PM

    ereming's Avatar

    Hi Chris,

    After some additional testing it seems that the Error depends on the update_interval parameter. Testing an older PCIe rig, the lower limit is between 0.5 and 0.75 ms for my particular channel configuration, whereas in the new Thunderbolt rig it's somewhere north of 3ms, the original setting. The official line I'm getting now from NI is that the PCIe-Thunderbolt adapter hasn't been tested and so isn't supported.

    Evan

  8. 8 Posted by ereming on 19 Feb, 2015 10:12 PM

    ereming's Avatar

    Hi Chris,

    Using your USB-6212, I'm getting the following error intermittently:

    ERROR: NIDAQ error: <err>Some or all of the samples requested have not yet been acquired.
    
    To wait for the samples to become available use a longer read timeout or read later in your program. To make the samples available sooner, increase the sample rate. (-200284)
    

    I've only tested update intervals of 2 and 3 ms and it happened for both. Surprisingly, when I moved the unit, I started getting the error in rapid succession.

  9. Support Staff 9 Posted by Christopher Sta... on 20 Feb, 2015 03:19 PM

    Christopher Stawarz's Avatar

    Hi Evan,

    I see that error sometimes, too. It seems like it's been showing up in my automated test results more frequently since I upgraded the test machine to NI-DAQmx Base 14.0. When you're done with the device, I'll run some tests and see what I can figure out.

    Surprisingly, when I moved the unit, I started getting the error in rapid succession.

    You mean, you saw lots of errors while you had the device in your hands and were physically moving it? That's really strange (and suggests it may be a hardware issue, not a software one).

    Chris

  10. Support Staff 10 Posted by Christopher Sta... on 24 Feb, 2015 08:17 PM

    Christopher Stawarz's Avatar

    Hi Evan,

    I've run some analog I/O tests with the USB-6212 on several different machines. Unfortunately, they haven't revealed much.

    On my primary test machine (late-2009 Mac mini running OS X 10.8), I can easily induce the "samples requested have not yet been acquired" error by loading up the CPU cores (e.g. making a couple Python processes evaluate infinite loops) or doing UI-intensive tasks in other applications (e.g. mucking around with Google Maps in Safari). This isn't too surprising, as that Mac is old and underpowered, and it's not hard to imagine things falling behind when the system is taxed.

    On a secondary test machine (late-2013 iMac running OS X 10.10), I can occasionally induce the error with similar activities. On my main development machine (mid-2010 Mac Pro running OS X 10.10), I can't induce the error at all.

    In addition to testing with MWorks, I tried a standalone analog I/O test (based on the example programs distributed with NI-DAQmx Base) and got the same results. I also tried downgrading the Mac mini to NI-DAQmx Base 3.7, but the results were unchanged.

    None of this suggests why you might be seeing these errors on your new Mac Pro. If the issue comes down to system load, then, if anything, your machine ought to be outperforming all the ones I tried.

    Would it possible for me to run some of my tests on your system? In particular, I'd like to try the non-MWorks test program. If I can reliably induce errors running that, then I think we'd have cause for opening another support request with NI.

    Thanks,
    Chris

  11. 11 Posted by ereming on 02 Mar, 2015 10:07 PM

    ereming's Avatar

    Hi Chris,

    I tried to reproduce the error on my older Mac Pro and so far haven't been able to see it unless I set the update interval to 1ms. Even then, it doesn't just start at some random delay then snowball out of control - it starts immediately.

    It's curious to me that the errors from the two different devices are different. The PCIe devices (Thunderbolt or no) give a "timeout" error, whereas the USB devices give the "samples have not been acquired error." This is constant across the computer being tested.

    Thanks,
    Evan

  12. Support Staff 12 Posted by Christopher Sta... on 03 Mar, 2015 09:24 PM

    Christopher Stawarz's Avatar

    Hi Evan,

    I tried to reproduce the error on my older Mac Pro and so far haven't been able to see it unless I set the update interval to 1ms. Even then, it doesn't just start at some random delay then snowball out of control - it starts immediately.

    Hmm. I suppose it's possible that the issue is unique to the new Mac Pro -- maybe a driver issue? However, it's not immediately clear how we could test that hypothesis. Let me think about it a bit.

    It's curious to me that the errors from the two different devices are different. The PCIe devices (Thunderbolt or no) give a "timeout" error, whereas the USB devices give the "samples have not been acquired error."

    That could just reflect that different driver pathways for PCIe and USB devices.

    Also, it's possible that the Thunderbolt problem is a different issue entirely. Have you tried the PCIe card + Thunderbolt chassis on any other Macs? I can try it on an iMac or Mac mini, if you like. (The old Mac Pro doesn't support Thunderbolt, so that's not an option.)

    Chris

  13. 13 Posted by ereming on 04 Mar, 2015 02:38 PM

    ereming's Avatar

    Hi Chris,

    Just to update from the last post - I let the program run for a day on the old mac pro using the USB DAQ and a 2 ms update interval and got a total of 3 errors in 24 hours (no consecutive errors). I also ran with the Thunderbolt expansion PCIe DAQ on a Macbook Air and only got one warning. You don't have a new Mac Pro in the lab that I could test do you?

    Evan

  14. Support Staff 14 Posted by Christopher Sta... on 04 Mar, 2015 03:05 PM

    Christopher Stawarz's Avatar

    Hi Evan,

    I also ran with the Thunderbolt expansion PCIe DAQ on a Macbook Air and only got one warning.

    That's really interesting. Maybe it is an issue with the new Mac Pro.

    You don't have a new Mac Pro in the lab that I could test do you?

    No, I don't think anyone in our lab has one yet.

    Chris

  15. Support Staff 15 Posted by Christopher Sta... on 04 Mar, 2015 03:18 PM

    Christopher Stawarz's Avatar

    I suppose it's possible that the issue is unique to the new Mac Pro -- maybe a driver issue? However, it's not immediately clear how we could test that hypothesis.

    Maybe the right way to proceed is for me to develop a simple, standalone application that performs the exact same I/O tasks as your MWorks experiment. If we can get that application to produce the errors you've been seeing with the USB DAQ, then you can open a support request with NI and ask them to check it out. The latest NI-DAQmx Base is supposed to support the USB-6212 on OS X 10.9, so if there are problems using it with the new Mac Pro, then that's definitely something they should address.

    Does this sound OK to you? If so, I'll pick up the DAQ sometime today.

    Chris

  16. 16 Posted by ereming on 04 Mar, 2015 03:44 PM

    ereming's Avatar

    I opened a service request back in January and I believe NI already tested the specific scenario directly after I sent them a copy of the experiment I'm using to test the issue. I never had them test it with a 6212 though, so maybe I can pick up that old thread with NI.

    I'd like to try and track down another Mac Pro here to see whether I can reproduce the problem. If not, I'll try a clean install on the computer and if that doesn't fix it, then it must be some hardware issue with our particular computer.

  17. Support Staff 17 Posted by Christopher Sta... on 13 Mar, 2015 03:30 PM

    Christopher Stawarz's Avatar

    Hi Evan,

    As we've discussed offline, I've been able to reproduce the "samples requested have not yet been acquired" error with the USB-6212, using both your MWorks experiment and a simple (non-MWorks) test application. I've seen the errors on a late 2012 Mac mini (running OS X 10.9.5) and a late 2013 iMac (running OS X 10.10.2). However, I still haven't seen them on my mid-2010 Mac Pro.

    I've submitted a support request to NI. (If you want to reference it in your communications with them, it's Service Request #7441619.) I'll let you know what I hear.

    Cheers,
    Chris

  18. 18 Posted by ereming on 13 Mar, 2015 04:31 PM

    ereming's Avatar

    Thanks for the update. I also got the same error on a mid-2014 Macbook Pro, but still haven't seen the error on our early 2014 Macbook air or mid 2012 Mac Pros.

  19. Support Staff 19 Posted by Christopher Sta... on 20 Mar, 2015 08:18 PM

    Christopher Stawarz's Avatar

    Hi Evan,

    Following up on our earlier (offline) discussion: I've updated and regenerated the MWorks nightly build so that the NIDAQ interface is once again built against NI-DAQmx Base 3.7. It'd be great if you could use the new build to re-test your PCIe card over Thunderbolt.

    FYI, I found that installing NI-DAQmx Base 3.7 over 14.0 left things in a bad state (e.g. my test app crashed almost immediately after starting), so you may want to remove all the 14.0 files before installing 3.7. To do this, open Terminal and run the following commands:

    sudo launchctl unload /Library/LaunchDaemons/nilxid.plist
    sudo rm -rf /Applications/National\ Instruments/ /Library/Frameworks/NiSpyLog.framework /Library/Frameworks/LabVIEW\ * /Library/Frameworks/lib* /Library/Application\ Support/National\ Instruments/ /Library/Frameworks/nidaqmxbase* /Library/Extensions/niusb9162k.kext/ /Library/Preferences/NIvisa/ /Library/Frameworks/VISA.framework/ /Library/LaunchDaemons/nilxid.plist /Library/Extensions/NiViPciK.kext/ /Library/Extensions/nipalk.kext/ /Library/StartupItems/nipal/ /usr/sbin/nipalsm /usr/sbin/palModuleMgr.sh /Library/Frameworks/NI-RPC.framework/ /Library/Frameworks/nisysapi.framework/
    

    Also, on systems running Yosemite, you'll need to disable the signing requirement for kernel extensions. In Terminal, run the command

    sudo nvram boot-args="kext-dev-mode=1"
    

    and then restart. (This isn't necessary on Mavericks or Mountain Lion.)

    Chris

  20. 20 Posted by ereming on 22 Mar, 2015 05:34 PM

    ereming's Avatar

    Hi Chris,

    Just want to make sure - the new nightly won't work with NI DAQ mx base 14
    anymore, only 3.7, right? If so I'll make sure the rest of the lab gets a
    heads up.

    - Evan

  21. Support Staff 21 Posted by Christopher Sta... on 23 Mar, 2015 01:15 PM

    Christopher Stawarz's Avatar

    Just want to make sure - the new nightly won't work with NI DAQ mx base 14 anymore, only 3.7, right?

    Yes, that's correct.

    Chris

  22. Support Staff 22 Posted by Christopher Sta... on 23 Mar, 2015 07:07 PM

    Christopher Stawarz's Avatar

    Hi Evan,

    After downgrading all my test systems to NI-DAQmx Base 3.7, I'm no longer seeing the "samples requested have not yet been acquired" errors with the USB-6212. (Well, I did see a couple instances on some of the test machines, but nothing like the massive spew I had been getting. I assume the remaining errors are due to system load.) If it turns out that the PCIe+Thunderbolt rig still has issues under 3.7, then the USB DAQ should be a workable alternative.

    Cheers,
    Chris

  23. 23 Posted by ereming on 25 Mar, 2015 05:10 PM

    ereming's Avatar

    I'm no longer getting the errors on the Mac Pro after downgrading to
    NI-DAQmx Base 3.7. I'm still getting warnings about skipping 10-20
    executions every 5 minutes or so with the USB, and every 15 minutes with
    the Thunderbolt-PCIe. Do you get these on your Mac Pro? I don't remember
    getting them on ours.

  24. Support Staff 24 Posted by Christopher Sta... on 27 Mar, 2015 02:51 PM

    Christopher Stawarz's Avatar

    I'm still getting warnings about skipping 10-20 executions every 5 minutes or so with the USB, and every 15 minutes with the Thunderbolt-PCIe. Do you get these on your Mac Pro?

    I see those sometimes (although not at those specific intervals, I think). Those warnings come from MWorks, not NI-DAQmx Base. Presumably, they're caused by one or more threads getting backed up due to system load or some other scheduling issue. With NIDAQ devices in particular, data has to hop through multiple processes and threads before it reaches MWorks; the delays could be happening anywhere along the route.

    My feeling is that this isn't something you need to worry about. Of course, if it seems like it's becoming a problem, I'll try to investigate what's happening.

    Chris

  25. Christopher Stawarz closed this discussion on 14 Apr, 2015 05:00 PM.

  26. Christopher Stawarz re-opened this discussion on 02 Mar, 2018 04:25 PM

  27. Support Staff 25 Posted by Christopher Sta... on 02 Mar, 2018 04:25 PM

    Christopher Stawarz's Avatar

    Hi Evan,

    After many weeks of back and forth with NI, we've finally reached some resolution on the "samples requested have not yet been acquired" error with USB devices.

    The NI applications engineer that I worked with (Clint, who was very helpful and really stuck with it despite the length of the process) was able, using my sample code, to reproduce the issue with a USB-6212 and NI-DAQmx Base 15.0, on a Mac with USB 3 running OS X 10.10. (Interestingly, he was not able to reproduce it on an older Mac with only USB 2, which lends some support to my hypothesis that USB 3 is involved somehow.) After that, Clint escalated the issue to NI's R&D department. After examining my test code, they provided him with their best guess of what's happening.

    To summarize: The process of acquiring analog samples on a USB DAQ and getting them in to a Mac application involves three different "actors". The first is the DAQ itself, which acquires the samples and stores them in on-board memory. The second is a software component, running on the Mac but not under the control of the end-user's software (e.g. MWorks or my test code), which periodically pulls new samples off the DAQ hardware and stores them in the Mac's memory. The third is the end-user application, which periodically uses NI-DAQmx Base functions to request new samples from the second component (not directly from the DAQ).

    The issues arise in the second component. Although it's doing its best to pull samples from the DAQ in a regular, timely fashion, it's ultimately just another process running on a time-sharing, non-realtime operating system. While it is, on average, capable of keeping up with the stream of incoming data, there are times when (due to CPU load, USB bus traffic, or some other resource allocation issue) it gets behind. It will catch up (if it didn't, you'd eventually get a distinct, buffer-overflow error due to the DAQ's onboard memory filling up), but during the intervals where it's behind, requests for new samples from the end-user application will fail.

    For example, my test code uses a sampling rate of 1kHz and tries to read 3 samples every 3ms. Between one read attempt and the next, the DAQ has definitely acquired 3 new samples. However, the "middle man" process may not have had a chance to pull those samples. Therefore, when the test code requests new samples, the middle layer reports that they aren't available.

    Unfortunately, there's really no solution that eliminates this behavior. You can avoid the errors by increasing the read timeout. (In my test code, the initial timeout was 3ms. Increasing it by just 20% appears to eliminate the errors, at least over a 5-minute run.) However, you still have to live with the fact that samples are sometimes going to come in later than you'd expect.

    A few questions remain unanswered:

    1. Why don't we see this issue with NI-DAQmx Base 3.7?
    2. How and why is USB 3 involved (if it is at all)?

    Clint wasn't able to provide much insight into these. Also, I don't think any of this applies to PCIe devices, which presumably use DMA or some other high-speed transfer mechanism to get samples off the DAQ.

    Given this information, I plan to add an analog read timeout parameter to MWorks' NI-DAQ interface. Beyond that, though, I'm not sure how we should proceed.

    It still seems like, at this point, NI only just barely supports macOS, so it's hard to see them as a reliable supplier of MWorks-compatible DAQ hardware in the long term. Also, the "middle man" approach that NI uses for data transfer from USB DAQ's isn't the only way to do things. For example, with LabJack devices, the code for pulling samples over USB runs in the end-user application, which may result in better overall performance (since there's one less software component that needs to be scheduled and run by the OS).

    For the record, the reference number for the support request that was just closed is 3109950.

    Cheers,
    Chris

  28. 26 Posted by ereming on 02 Mar, 2018 05:57 PM

    ereming's Avatar

    Hi Chris,

    Thank you for working to sort this out. This will be great information to
    have when making hardware decisions going forward.

    Best,
    Evan

  29. Support Staff 27 Posted by Christopher Sta... on 27 Mar, 2018 07:51 PM

    Christopher Stawarz's Avatar

    Hi Evan,

    I plan to add an analog read timeout parameter to MWorks' NI-DAQ interface.

    This is done and available in the nightly build. The parameter is called analog_read_timeout. If omitted, it defaults to the value of update_interval (which preserves the old behavior).

    Cheers,
    Chris

  30. Christopher Stawarz closed this discussion on 03 Apr, 2018 02:10 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac