ROS2 Support for the ZA6

arbogastaw · April 10, 2025, 12:34pm

The EoL for ROS Noetic is scheduled May 31st, 2025. I’m curios if there are plans to support the ZA6 in ROS2 and what the timeline for that migration might look like.

I believe hal_ros_control has had ROS2 support for over 2 years. It’s been a while since I have updated my Pathpilot images, but it looks like these still contain the Noetic tag in the image name.

Thanks!

futnuh · June 1, 2025, 12:45am

@arbogastaw I’m curious if Tormach got back to you about this?

arbogastaw · June 2, 2025, 2:55pm

I have not heard back yet, but still hoping for ROS2 support in the near future.

John_Morris · June 13, 2025, 3:32pm

Hi Alex, thanks for posting this! After we saw your question, I cleaned up and pushed our initial ROS 2 support git repo, including robot hardware drivers and MoveIt2 configuration, but without the Tormach robot UI and TRPL interpreter.

We developed this some time back, in part, with customers in mind who don’t care as much about the UI, and just want direct access to the MoveIt2 or ros_control layers. Hope it’s useful to you.

arbogastaw · June 13, 2025, 3:50pm

@John_Morris This looks exactly like what I was looking for. Thank you! I will give this a shot on our system and report back with any success/issues we have along the way.

futnuh · July 23, 2025, 12:54am

@John_Morris Thanks for making the ROS2 drivers available for download. It’s really appreciated. (Cool to see Tormach listed as a supported platform on the MoveIt Pro site as well).

I’m struggling a bit trying to build the new container as detailed on GitHub - tormach/tormach_za_ros2_drivers: Tormach ZA robot ROS2 `hal_ros_control` and MoveIt2 configuration.

I’m doing this on the Ubuntu machine that ships with the ZA6, i.e.,
Linux pathpilot-controller 5.10.41-rt42 #1 SMP PREEMPT_RT Thu Jun 24 09:30:19 CDT 2021 x86_64 x86_64 x86_64 GNU/Linux

The docker-dev.sh -b script is failing as follows:

#16 58.77 Setting up curl (7.81.0-1ubuntu1.20) ...
#16 58.78 Processing triggers for libc-bin (2.35-0ubuntu3.10) ...
#16 59.18 + test ubuntu = debian
#16 59.18 + DOCKER_KEYRING=/usr/share/keyrings/docker.gpg
#16 59.18 + DOCKER_REPO=https://download.docker.com/linux/ubuntu
#16 59.18 + curl -fsSL https://download.docker.com/linux/ubuntu/gpg
#16 59.18 + gpg --dearmor
#16 59.22 curl: (6) getaddrinfo() thread failed to start
#16 59.22 gpg: no valid OpenPGP data found.
#16 ERROR: executor failed running [/bin/sh -c base/1-system.sh]: exit code: 2
------
 > [base 10/10] RUN --mount=type=cache,target=/var/cache/apt     base/1-system.sh:
------
executor failed running [/bin/sh -c base/1-system.sh]: exit code: 2

That curl command passes successfully in the terminal so I’m unsure what could be causing it to fail when launched in the script. I’m wondering if perhaps I’ve hit some sort of thread or memory limit? When you run this script, did you build it on actual ZA6 controller or on a separate dev machine?
Any idea what might be wrong?

John_Morris · July 24, 2025, 8:45pm

Hi @futnuh,

I confess I hadn’t built it on the robot controller for quite a while, and ran into the same problem you did when I tried.

Went down some rabbitholes trying to figure out what was happening, turns out to be a permission denied error in a clone3() syscall in curl, which wants CAP_SYS_ADMIN privs. Usually oddball permission error in docker build are from some container OS making some update that breaks stuff until Docker Engine gets an update to fix it; that was indeed the fix on my controller.

So, update the docker-ce & related packages:

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin

In case the Docker repo isn’t configured on your controller like mine, you can follow the instructions in the “Install using the apt repository” section of the following link. (I put the Docker pkg signing key in /usr/share/keyrings/docker.asc instead, alongside other keys, since /etc/apt/keyrings/ doesn’t exist.)

And before you build the image on the controller, be sure to stop the EtherCAT master, or the image build will break:

sudo systemctl stop ethercat

futnuh · July 26, 2025, 3:58am

@John_Morris Thanks John, I really appreciate you taking the time to investigate. That indeed sorted out the build issue. I managed to get RViz up and running in sim mode. However, when trying to connect to the actual hardware, hal_mgr encounters errors.

[ERROR] [hal_hw_interface.launch.hal_files]: Error loading HAL file za6.hal.py:
[ERROR] [hal_hw_interface.launch.hal_files]: Traceback (most recent call last):
  File "/opt/ros/humble/local/lib/python3.10/dist-packages/hal_hw_interface/launch/hal_files.py", line 102, in execute_deferred
    exec(data, globals_)
  File "/home/pathpilot/tormach_za_ros2_ws/install/za6_hardware/share/za6_hardware/halfiles/za6.hal.py", line 56, in <module>
    raise e
  File "/home/pathpilot/tormach_za_ros2_ws/install/za6_hardware/share/za6_hardware/halfiles/za6.hal.py", line 53, in <module>
    plumber_class(params).setup_hal()
  File "/home/pathpilot/tormach_za_ros2_ws/install/za6_hardware/local/lib/python3.10/dist-packages/hal_plumber/__init__.py", line 99, in setup_hal
    self.setup_drive()
  File "/home/pathpilot/tormach_za_ros2_ws/install/za6_hardware/local/lib/python3.10/dist-packages/hal_plumber/__init__.py", line 359, in setup_drive
    rtapi.loadrt("lcec")
  File "hal/cython/machinekit/rtapi.pyx", line 219, in machinekit.rtapi.RTAPIcommand.loadrt
RuntimeError: rtapi_loadrt '('lcec',)' failed: do_load_cmd: /usr/lib/machinekit/modules/lcec.so: undefined symbol: LCEC_PDOINIT

This symbol isn’t defined in the installed lcec.so:

$ nm -D /usr/lib/machinekit/modules/lcec.so | grep LCEC
                 U LCEC_PDOINIT

For reference, here is the entire log output from trying to run the launch script with added parameter hal_debug_level:=5. Note, I had to rename the uploaded file to .nc to get past the forum restrictions … but it’s just plain text.

paste3.txt.nc (138.2 KB)

John_Morris · July 31, 2025, 11:10pm

When I was publishing this stuff, I fumbled around with linuxcnc-ethercat branches, and got the wrong one. Very sorry about that.

Can you try the branch in this pull request? I have built it, on a Tormach robot controller, and verified the robot controls do launch correctly.

github.com/tormach/tormach_za_ros2_drivers

Zultron/2025 07 31 fix lcec symbols

humble ← zultron:zultron/2025-07-31_fix_lcec_symbols

opened 11:02PM - 31 Jul 25 UTC

zultron

+18 -3

User futnuh on the Tormach robot forums pointed out a linker problem loading the… `lcec` component because of a missing symbol, `LCEC_PDOINIT`. https://forums.tormach.com/t/ros2-support-for-the-za6/2370/8 That's now fixed in this branch: https://github.com/zultron/linuxcnc-ethercat/tree/zultron/2025-07-31-fix_linker_error ...along with the bit-rotted GH Actions config that pushes the packages here: https://cloudsmith.io/~zultron/repos/etherlabmaster-test/packages/ This PR bumps an env var in the `Dockerfile` that invalidates cache and forces the new version of the package to be installed. It also prints an error and exits the Docker build if the EtherCAT master is running, which causes the Docker image build to fail with only cryptic error messages.

Thanks for testing this out, @futnuh, I very much appreciate it!

futnuh · August 3, 2025, 5:47am

I applied the pull request and built the docker container(s). Running ros2-devel, I can now successfully start without having to be in sim mode

ros2 launch za6_bringup bringup.launch hal_debug_level:=5

However, when I try to engage the drives with the following,

$ ros2 service call /enable_drives std_srvs/srv/Trigger
requester: making request: std_srvs.srv.Trigger_Request()

There is no “click” from the robot. Over in the main shell, I get:

[drive_state-10] [INFO 1754199045.890323101] [drive_state]: /enable_drives service called (enable_svc_cb() at /home/pathpilot/tormach_za_ros2_ws/install/za6_hardware/local/lib/python3.10/dist-packages/hal_plumber/drive_state.py:320)
[hal_mgr-1] [ERROR 1754199045.891227706] [joint_trajectory_controller]: Trajectories with effort fields are currently not supported. (validate_trajectory_msg() at ./src/joint_trajectory_controller.cpp:1559)
[drive_state-10] [INFO 1754199045.891421080] [drive_state]: Published trajectory to zero command error (zero_error() at /home/pathpilot/tormach_za_ros2_ws/install/za6_hardware/local/lib/python3.10/dist-packages/hal_plumber/drive_state.py:241)
[drive_state-10] [ERROR 1754199055.892945913] [drive_state]: 'start' command timeout after 10.000000091999937 s in state IDLE. (check_timeout() at /home/pathpilot/tormach_za_ros2_ws/install/za6_hardware/local/lib/python3.10/dist-packages/hal_plumber/drive_state.py:212)
[drive_state-10] [ERROR 1754199055.894093460] [drive_state]: Zero command-feedback error timed out (zero_error() at /home/pathpilot/tormach_za_ros2_ws/install/za6_hardware/local/lib/python3.10/dist-packages/hal_plumber/drive_state.py:257)

After a while, the “enable_drives” command times out with

response:
std_srvs.srv.Trigger_Response(success=False, message="'start' command timeout after 10.000000091999937 s in state IDLE.")

For reference, the full console output from the “main” shell is attached below.
paste4.txt.nc (114.9 KB)

I’ll look at this this with fresh eyes tomorrow (currently 11pm and I’m at work). But if you have any thoughts on further debugging, I appreciate the guidance.

futnuh · August 6, 2025, 3:27am

I’m stuck.

The good news is that I can interact with the digital i/o on the robot directly from ros2 (i.e. publishing a properly crafted message to the appropriate topic). Separately, RViz is showing the real joint configuration of the robot on startup. That would indicate to me that Hal is correctly retrieving the joint positions from the servos.

But I cannot get the drives themselves to energize. And as described above, the only log messages (even with hal_debug_level:=5) that show up when trying to call the /enable_drives service are:

[drive_state-10] [INFO 1754199045.890323101] [drive_state]: /enable_drives service called (enable_svc_cb() at /home/pathpilot/tormach_za_ros2_ws/install/za6_hardware/local/lib/python3.10/dist-packages/hal_plumber/drive_state.py:320)
[hal_mgr-1] [ERROR 1754199045.891227706] [joint_trajectory_controller]: Trajectories with effort fields are currently not supported. (validate_trajectory_msg() at ./src/joint_trajectory_controller.cpp:1559)
[drive_state-10] [INFO 1754199045.891421080] [drive_state]: Published trajectory to zero command error (zero_error() at /home/pathpilot/tormach_za_ros2_ws/install/za6_hardware/local/lib/python3.10/dist-packages/hal_plumber/drive_state.py:241)
[drive_state-10] [ERROR 1754199055.892945913] [drive_state]: 'start' command timeout after 10.000000091999937 s in state IDLE. (check_timeout() at /home/pathpilot/tormach_za_ros2_ws/install/za6_hardware/local/lib/python3.10/dist-packages/hal_plumber/drive_state.py:212)
[drive_state-10] [ERROR 1754199055.894093460] [drive_state]: Zero command-feedback error timed out (zero_error() at /home/pathpilot/tormach_za_ros2_ws/install/za6_hardware/local/lib/python3.10/dist-packages/hal_plumber/drive_state.py:257)

Question, is it possibly something to do with the fact that I have one of the first versions of the ZA6 (serial number/date ZA 10003/2020-05)?

John_Morris · August 6, 2025, 2:35pm

Updated PR with fixes for more issues.

I fixed the “Trajectories with effort fields are currently not supported” problem.

There was also a bug we fixed in Machinekit HAL that needed changes in a HAL comp from this repo.

This seems to run smoothly on my robot, so I’m optimistic for you. Give it a try.

Your robot is indeed a very early one, but it should be supported, and if it isn’t, I’ll work with you to get it there. Thanks for bearing with me on this. The ROS 2 code isn’t well tested at all, just me, and now you!

futnuh · August 7, 2025, 2:27am

Hi John. I applied the update, rebuilt the container, and ran. Unfortunately, when trying to /enable_drives, this service call is still timing out (with no unlocking of the servo brakes heard):

$ ros2 service call /enable_drives std_srvs/srv/Trigger
requester: making request: std_srvs.srv.Trigger_Request()

response:
std_srvs.srv.Trigger_Response(success=False, message="'start' command timeout after 10.016535877999956 s in state IDLE.")

Over in the main logs, there are a few errors and warnings cropping up. If I filter the full log by [hw_device_mgr-8], we get a good look at the hardware startup. Everything looks good until around line 883. (Note this is before the /enable_drives service gets called in another shell, around line 986.

270:[INFO] [hw_device_mgr-8]: process details: cmd='/home/pathpilot/tormach_za_ros2_ws/install/za6_hardware/lib/za6_hardware/hw_device_mgr --ros-args --params-file /tmp/launch_params_l3qu1cf1 --params-file /home/pathpilot/tormach_za_ros2_ws/install/za6_hardware/share/za6_hardware/config/hal_hw_device_mgr.yaml', cwd='None', custom_env?=True
271:[INFO] [hw_device_mgr-8]: process started with pid [514]
304:[hw_device_mgr-8] 2025-08-06 17:37:41,029 [INFO]hw_device_mgr.ethercat.xml_reader: Reading ESI from (hw_device_mgr.devices.device_xml, IS620N_v2.6.7.xml)
305:[hw_device_mgr-8] 2025-08-06 17:37:41,077 [INFO]hw_device_mgr.ethercat.xml_reader: Reading ESI from (hw_device_mgr.devices.device_xml, BogusServo.xml)
308:[hw_device_mgr-8] 2025-08-06 17:37:41,085 [INFO]hw_device_mgr.ethercat.xml_reader: Reading ESI from (hw_device_mgr.devices.device_xml, SV660_EOE_1Axis_V9.12.xml)
309:[hw_device_mgr-8] 2025-08-06 17:37:41,128 [INFO]hw_device_mgr.ethercat.xml_reader: Reading ESI from (hw_device_mgr.devices.device_xml, iTegva_E7x_Series.xml)
310:[hw_device_mgr-8] 2025-08-06 17:37:41,152 [INFO]ZA_device_mgr: Starting in hardware mode
311:[hw_device_mgr-8] 2025-08-06 17:37:41,152 [INFO]ZA_device_mgr: Initializing 'ZA_device_mgr' ROS node
312:[hw_device_mgr-8] 2025-08-06 17:37:41,153 [INFO]ZA_device_mgr: Reading device config from '/home/pathpilot/tormach_za_ros2_ws/install/za6_hardware/share/za6_hardware/config/hal_device_config.yaml'
313:[hw_device_mgr-8] 2025-08-06 17:37:41,334 [INFO]ZA_device_mgr: New device manager instance starting
314:[hw_device_mgr-8] 2025-08-06 17:37:41,334 [INFO]ZA_device_mgr: Initializing devices
315:[hw_device_mgr-8] 2025-08-06 17:37:41,338 [INFO](0x00100000, 0x000C0108)@(0,0,0): LCECConfig:(0x00100000,0x000C0108)@(0,0,0) created from bus scan
316:[hw_device_mgr-8] 2025-08-06 17:37:41,338 [INFO](0x00100000, 0x000C0108)@(0,1,0): LCECConfig:(0x00100000,0x000C0108)@(0,1,0) created from bus scan
317:[hw_device_mgr-8] 2025-08-06 17:37:41,339 [INFO](0x00100000, 0x000C0108)@(0,2,0): LCECConfig:(0x00100000,0x000C0108)@(0,2,0) created from bus scan
318:[hw_device_mgr-8] 2025-08-06 17:37:41,339 [INFO](0x00100000, 0x000C0108)@(0,3,0): LCECConfig:(0x00100000,0x000C0108)@(0,3,0) created from bus scan
319:[hw_device_mgr-8] 2025-08-06 17:37:41,339 [INFO](0x00100000, 0x000C0108)@(0,4,0): LCECConfig:(0x00100000,0x000C0108)@(0,4,0) created from bus scan
320:[hw_device_mgr-8] 2025-08-06 17:37:41,339 [INFO](0x00100000, 0x000C0108)@(0,5,0): LCECConfig:(0x00100000,0x000C0108)@(0,5,0) created from bus scan
321:[hw_device_mgr-8] 2025-08-06 17:37:41,339 [INFO](0x00000A09, 0x00000201)@(0,6,0): LCECConfig:(0x00000A09,0x00000201)@(0,6,0) created from bus scan
326:[hw_device_mgr-8] 2025-08-06 17:37:41,405 [INFO]ZA_device_mgr: Initialized device ZA_IS620N@(0,0,0)
327:[hw_device_mgr-8] 2025-08-06 17:37:41,406 [INFO]ZA_device_mgr: Initialized device ZA_IS620N@(0,1,0)
328:[hw_device_mgr-8] 2025-08-06 17:37:41,407 [INFO]ZA_device_mgr: Initialized device ZA_IS620N@(0,2,0)
329:[hw_device_mgr-8] 2025-08-06 17:37:41,408 [INFO]ZA_device_mgr: Initialized device ZA_IS620N@(0,3,0)
330:[hw_device_mgr-8] 2025-08-06 17:37:41,408 [INFO]ZA_device_mgr: Initialized device ZA_IS620N@(0,4,0)
331:[hw_device_mgr-8] 2025-08-06 17:37:41,409 [INFO]ZA_device_mgr: Initialized device ZA_IS620N@(0,5,0)
332:[hw_device_mgr-8] 2025-08-06 17:37:41,410 [INFO]ZA_device_mgr: Initialized device ZA_E7.820.003@(0,6,0)
333:[hw_device_mgr-8] 2025-08-06 17:37:41,411 [INFO]ZA_device_mgr: Device manager initialization complete
334:[hw_device_mgr-8] 2025-08-06 17:37:41,416 [INFO]ZA_device_mgr: HAL component 'hw_device_mgr' ready
335:[hw_device_mgr-8] 2025-08-06 17:37:41,417 [INFO]ZA_device_mgr: Initialized 'ZA_device_mgr' ROS node
344:[hw_device_mgr-8] 2025-08-06 17:37:41,520 [INFO]ZA_IS620N@(0,0,0): Goal not reached:  Offline
345:[hw_device_mgr-8] 2025-08-06 17:37:41,521 [INFO]ZA_IS620N@(0,1,0): Goal not reached:  Offline
346:[hw_device_mgr-8] 2025-08-06 17:37:41,521 [INFO]ZA_IS620N@(0,2,0): Goal not reached:  Offline
347:[hw_device_mgr-8] 2025-08-06 17:37:41,521 [INFO]ZA_IS620N@(0,3,0): Goal not reached:  Offline
348:[hw_device_mgr-8] 2025-08-06 17:37:41,521 [INFO]ZA_IS620N@(0,4,0): Goal not reached:  Offline
349:[hw_device_mgr-8] 2025-08-06 17:37:41,521 [INFO]ZA_IS620N@(0,5,0): Goal not reached:  Offline
350:[hw_device_mgr-8] 2025-08-06 17:37:41,521 [INFO]ZA_E7.820.003@(0,6,0): Goal not reached:  Offline
351:[hw_device_mgr-8] 2025-08-06 17:37:41,521 [INFO]ZA_device_mgr: Goal not reached:  Offline (ZA_IS620N@(0,0,0),ZA_IS620N@(0,1,0),ZA_IS620N@(0,2,0),ZA_IS620N@(0,3,0),ZA_IS620N@(0,4,0),ZA_IS620N@(0,5,0),ZA_E7.820.003@(0,6,0))
352:[hw_device_mgr-8] 2025-08-06 17:37:41,522 [INFO]ZA_device_mgr: Waiting for devices to come online before init
825:[hw_device_mgr-8] 2025-08-06 17:37:43,517 [INFO]ZA_IS620N@(0,0,0): Drive came online
826:[hw_device_mgr-8] 2025-08-06 17:37:43,518 [INFO]ZA_IS620N@(0,1,0): Drive came online
827:[hw_device_mgr-8] 2025-08-06 17:37:43,518 [INFO]ZA_IS620N@(0,2,0): Drive came online
828:[hw_device_mgr-8] 2025-08-06 17:37:43,519 [INFO]ZA_IS620N@(0,3,0): Drive came online
829:[hw_device_mgr-8] 2025-08-06 17:37:43,519 [INFO]ZA_IS620N@(0,4,0): Drive came online
830:[hw_device_mgr-8] 2025-08-06 17:37:43,520 [INFO]ZA_IS620N@(0,5,0): Drive came online
831:[hw_device_mgr-8] 2025-08-06 17:37:43,520 [INFO]ZA_E7.820.003@(0,6,0): Drive came online
832:[hw_device_mgr-8] 2025-08-06 17:37:43,520 [INFO]ZA_IS620N@(0,0,0): Goal not reached:  Not operational, device params unset
833:[hw_device_mgr-8] 2025-08-06 17:37:43,520 [INFO]ZA_IS620N@(0,1,0): Goal not reached:  Not operational, device params unset
834:[hw_device_mgr-8] 2025-08-06 17:37:43,521 [INFO]ZA_IS620N@(0,2,0): Goal not reached:  Not operational, device params unset
835:[hw_device_mgr-8] 2025-08-06 17:37:43,521 [INFO]ZA_IS620N@(0,3,0): Goal not reached:  Not operational, device params unset
836:[hw_device_mgr-8] 2025-08-06 17:37:43,521 [INFO]ZA_IS620N@(0,4,0): Goal not reached:  Not operational, device params unset
837:[hw_device_mgr-8] 2025-08-06 17:37:43,521 [INFO]ZA_IS620N@(0,5,0): Goal not reached:  Not operational, device params unset
838:[hw_device_mgr-8] 2025-08-06 17:37:43,521 [INFO]ZA_E7.820.003@(0,6,0): Goal not reached:  Not operational, device params unset
839:[hw_device_mgr-8] 2025-08-06 17:37:43,521 [INFO]ZA_device_mgr: Goal not reached:  Not operational, device params unset (ZA_IS620N@(0,0,0),ZA_IS620N@(0,1,0),ZA_IS620N@(0,2,0),ZA_IS620N@(0,3,0),ZA_IS620N@(0,4,0),ZA_IS620N@(0,5,0),ZA_E7.820.003@(0,6,0))
840:[hw_device_mgr-8] 2025-08-06 17:37:43,521 [INFO]ZA_IS620N@(0,0,0): Initializing params after coming online
841:[hw_device_mgr-8] 2025-08-06 17:37:43,522 [INFO]ZA_IS620N_cfg@(0,0,0): Queueing 36 param updates
842:[hw_device_mgr-8] 2025-08-06 17:37:43,523 [INFO]ZA_IS620N@(0,1,0): Initializing params after coming online
843:[hw_device_mgr-8] 2025-08-06 17:37:43,524 [INFO]ZA_IS620N_cfg@(0,1,0): Queueing 36 param updates
844:[hw_device_mgr-8] 2025-08-06 17:37:43,525 [INFO]ZA_IS620N@(0,2,0): Initializing params after coming online
845:[hw_device_mgr-8] 2025-08-06 17:37:43,526 [INFO]ZA_IS620N_cfg@(0,2,0): Queueing 36 param updates
846:[hw_device_mgr-8] 2025-08-06 17:37:43,527 [INFO]ZA_IS620N@(0,3,0): Initializing params after coming online
847:[hw_device_mgr-8] 2025-08-06 17:37:43,528 [INFO]ZA_IS620N_cfg@(0,3,0): Queueing 36 param updates
848:[hw_device_mgr-8] 2025-08-06 17:37:43,529 [INFO]ZA_IS620N@(0,4,0): Initializing params after coming online
849:[hw_device_mgr-8] 2025-08-06 17:37:43,530 [INFO]ZA_IS620N_cfg@(0,4,0): Queueing 36 param updates
850:[hw_device_mgr-8] 2025-08-06 17:37:43,530 [INFO]ZA_IS620N@(0,5,0): Initializing params after coming online
851:[hw_device_mgr-8] 2025-08-06 17:37:43,531 [INFO]ZA_IS620N_cfg@(0,5,0): Queueing 36 param updates
852:[hw_device_mgr-8] 2025-08-06 17:37:43,619 [INFO]ZA_IS620N@(0,0,0): Goal not reached:  Not operational, updating device params
853:[hw_device_mgr-8] 2025-08-06 17:37:43,619 [INFO]ZA_IS620N@(0,1,0): Goal not reached:  Not operational, updating device params
854:[hw_device_mgr-8] 2025-08-06 17:37:43,619 [INFO]ZA_IS620N@(0,2,0): Goal not reached:  Not operational, updating device params
855:[hw_device_mgr-8] 2025-08-06 17:37:43,620 [INFO]ZA_IS620N@(0,3,0): Goal not reached:  Not operational, updating device params
856:[hw_device_mgr-8] 2025-08-06 17:37:43,620 [INFO]ZA_IS620N@(0,4,0): Goal not reached:  Not operational, updating device params
857:[hw_device_mgr-8] 2025-08-06 17:37:43,620 [INFO]ZA_IS620N@(0,5,0): Goal not reached:  Not operational, updating device params
858:[hw_device_mgr-8] 2025-08-06 17:37:43,620 [INFO]ZA_device_mgr: Goal not reached:  Not operational, updating device params (ZA_IS620N@(0,0,0),ZA_IS620N@(0,1,0),ZA_IS620N@(0,2,0),ZA_IS620N@(0,3,0),ZA_IS620N@(0,4,0),ZA_IS620N@(0,5,0)); Not operational, device params unset (ZA_E7.820.003@(0,6,0))
861:[hw_device_mgr-8] 2025-08-06 17:37:44,017 [INFO]ZA_IS620N@(0,0,0): status_word:  0x0250 SWITCH ON DISABLED flags: VOLTAGE_ENABLED,REMOTE
862:[hw_device_mgr-8] 2025-08-06 17:37:44,017 [INFO]ZA_IS620N@(0,1,0): status_word:  0x0250 SWITCH ON DISABLED flags: VOLTAGE_ENABLED,REMOTE
863:[hw_device_mgr-8] 2025-08-06 17:37:44,017 [INFO]ZA_IS620N@(0,2,0): status_word:  0x0250 SWITCH ON DISABLED flags: VOLTAGE_ENABLED,REMOTE
864:[hw_device_mgr-8] 2025-08-06 17:37:44,018 [INFO]ZA_IS620N@(0,3,0): status_word:  0x0250 SWITCH ON DISABLED flags: VOLTAGE_ENABLED,REMOTE
865:[hw_device_mgr-8] 2025-08-06 17:37:44,018 [INFO]ZA_IS620N@(0,4,0): status_word:  0x0250 SWITCH ON DISABLED flags: VOLTAGE_ENABLED,REMOTE
866:[hw_device_mgr-8] 2025-08-06 17:37:44,018 [INFO]ZA_IS620N@(0,5,0): status_word:  0x0250 SWITCH ON DISABLED flags: VOLTAGE_ENABLED,REMOTE
869:[hw_device_mgr-8] 2025-08-06 17:37:44,517 [INFO]ZA_IS620N@(0,0,0): Drive came online/operational
870:[hw_device_mgr-8] 2025-08-06 17:37:44,518 [INFO]ZA_IS620N@(0,1,0): Drive came online/operational
871:[hw_device_mgr-8] 2025-08-06 17:37:44,518 [INFO]ZA_IS620N@(0,2,0): Drive came online/operational
872:[hw_device_mgr-8] 2025-08-06 17:37:44,518 [INFO]ZA_IS620N@(0,3,0): Drive came online/operational
873:[hw_device_mgr-8] 2025-08-06 17:37:44,519 [INFO]ZA_IS620N@(0,4,0): Drive came online/operational
874:[hw_device_mgr-8] 2025-08-06 17:37:44,519 [INFO]ZA_IS620N@(0,5,0): Drive came online/operational
875:[hw_device_mgr-8] 2025-08-06 17:37:44,519 [INFO]ZA_E7.820.003@(0,6,0): Drive came online/operational
876:[hw_device_mgr-8] 2025-08-06 17:37:44,519 [INFO]ZA_IS620N@(0,0,0): Goal not reached:  updating device params
877:[hw_device_mgr-8] 2025-08-06 17:37:44,520 [INFO]ZA_IS620N@(0,1,0): Goal not reached:  updating device params
878:[hw_device_mgr-8] 2025-08-06 17:37:44,520 [INFO]ZA_IS620N@(0,2,0): Goal not reached:  updating device params
879:[hw_device_mgr-8] 2025-08-06 17:37:44,520 [INFO]ZA_IS620N@(0,3,0): Goal not reached:  updating device params
880:[hw_device_mgr-8] 2025-08-06 17:37:44,520 [INFO]ZA_IS620N@(0,4,0): Goal not reached:  updating device params
881:[hw_device_mgr-8] 2025-08-06 17:37:44,520 [INFO]ZA_IS620N@(0,5,0): Goal not reached:  updating device params
882:[hw_device_mgr-8] 2025-08-06 17:37:44,520 [INFO]ZA_E7.820.003@(0,6,0): Goal not reached:  device params unset
883:[hw_device_mgr-8] 2025-08-06 17:37:44,520 [INFO]ZA_device_mgr: Goal not reached:  updating device params (ZA_IS620N@(0,0,0),ZA_IS620N@(0,1,0),ZA_IS620N@(0,2,0),ZA_IS620N@(0,3,0),ZA_IS620N@(0,4,0),ZA_IS620N@(0,5,0)); device params unset (ZA_E7.820.003@(0,6,0))
896:[hw_device_mgr-8] 2025-08-06 17:37:46,617 [ERROR]ZA_device_mgr: Timeout (5s):  updating device params (ZA_IS620N@(0,0,0),ZA_IS620N@(0,1,0),ZA_IS620N@(0,2,0),ZA_IS620N@(0,3,0),ZA_IS620N@(0,4,0),ZA_IS620N@(0,5,0)); device params unset (ZA_E7.820.003@(0,6,0))
897:[hw_device_mgr-8] 2025-08-06 17:37:46,619 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
898:[hw_device_mgr-8] 2025-08-06 17:37:46,619 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
899:[hw_device_mgr-8] 2025-08-06 17:37:47,817 [INFO]ZA_IS620N@(0,0,0): Device param init complete
900:[hw_device_mgr-8] 2025-08-06 17:37:47,818 [INFO]ZA_IS620N@(0,1,0): Device param init complete
901:[hw_device_mgr-8] 2025-08-06 17:37:47,818 [INFO]ZA_IS620N@(0,2,0): Device param init complete
902:[hw_device_mgr-8] 2025-08-06 17:37:47,818 [INFO]ZA_IS620N@(0,3,0): Device param init complete
903:[hw_device_mgr-8] 2025-08-06 17:37:47,819 [INFO]ZA_IS620N@(0,4,0): Device param init complete
904:[hw_device_mgr-8] 2025-08-06 17:37:47,819 [INFO]ZA_IS620N@(0,5,0): Device param init complete
905:[hw_device_mgr-8] 2025-08-06 17:37:47,820 [INFO]ZA_IS620N@(0,0,0): Goal reached:  Reached
906:[hw_device_mgr-8] 2025-08-06 17:37:47,820 [INFO]ZA_IS620N@(0,1,0): Goal reached:  Reached
907:[hw_device_mgr-8] 2025-08-06 17:37:47,820 [INFO]ZA_IS620N@(0,2,0): Goal reached:  Reached
908:[hw_device_mgr-8] 2025-08-06 17:37:47,820 [INFO]ZA_IS620N@(0,3,0): Goal reached:  Reached
909:[hw_device_mgr-8] 2025-08-06 17:37:47,821 [INFO]ZA_IS620N@(0,4,0): Goal reached:  Reached
910:[hw_device_mgr-8] 2025-08-06 17:37:47,821 [INFO]ZA_IS620N@(0,5,0): Goal reached:  Reached
911:[hw_device_mgr-8] 2025-08-06 17:37:47,821 [INFO]ZA_device_mgr: Goal not reached:  device params unset (ZA_E7.820.003@(0,6,0))
912:[hw_device_mgr-8] 2025-08-06 17:37:51,817 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
913:[hw_device_mgr-8] 2025-08-06 17:37:51,819 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
914:[hw_device_mgr-8] 2025-08-06 17:37:51,819 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
915:[hw_device_mgr-8] 2025-08-06 17:37:57,117 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
916:[hw_device_mgr-8] 2025-08-06 17:37:57,118 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
917:[hw_device_mgr-8] 2025-08-06 17:37:57,118 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
918:[hw_device_mgr-8] 2025-08-06 17:38:02,317 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
919:[hw_device_mgr-8] 2025-08-06 17:38:02,318 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
920:[hw_device_mgr-8] 2025-08-06 17:38:02,319 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
921:[hw_device_mgr-8] 2025-08-06 17:38:07,517 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
922:[hw_device_mgr-8] 2025-08-06 17:38:07,518 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
923:[hw_device_mgr-8] 2025-08-06 17:38:07,518 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
924:[hw_device_mgr-8] 2025-08-06 17:38:11,618 [ERROR]ZA_E7.820.003@(0,6,0): Timeout (30s):  device params unset
925:[hw_device_mgr-8] 2025-08-06 17:38:11,618 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
926:[hw_device_mgr-8] 2025-08-06 17:38:11,618 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
927:[hw_device_mgr-8] 2025-08-06 17:38:16,917 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
928:[hw_device_mgr-8] 2025-08-06 17:38:16,918 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
929:[hw_device_mgr-8] 2025-08-06 17:38:16,918 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
930:[hw_device_mgr-8] 2025-08-06 17:38:22,117 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
931:[hw_device_mgr-8] 2025-08-06 17:38:22,119 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
932:[hw_device_mgr-8] 2025-08-06 17:38:22,119 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
933:[hw_device_mgr-8] 2025-08-06 17:38:27,417 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
934:[hw_device_mgr-8] 2025-08-06 17:38:27,418 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
935:[hw_device_mgr-8] 2025-08-06 17:38:27,418 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
936:[hw_device_mgr-8] 2025-08-06 17:38:32,617 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
937:[hw_device_mgr-8] 2025-08-06 17:38:32,619 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
938:[hw_device_mgr-8] 2025-08-06 17:38:32,619 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
939:[hw_device_mgr-8] 2025-08-06 17:38:37,917 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
940:[hw_device_mgr-8] 2025-08-06 17:38:37,918 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
941:[hw_device_mgr-8] 2025-08-06 17:38:37,918 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
942:[hw_device_mgr-8] 2025-08-06 17:38:41,918 [ERROR]ZA_E7.820.003@(0,6,0): Timeout (30s):  device params unset
943:[hw_device_mgr-8] 2025-08-06 17:38:41,918 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
944:[hw_device_mgr-8] 2025-08-06 17:38:41,918 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
945:[hw_device_mgr-8] 2025-08-06 17:38:47,217 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
946:[hw_device_mgr-8] 2025-08-06 17:38:47,218 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
947:[hw_device_mgr-8] 2025-08-06 17:38:47,219 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
948:[hw_device_mgr-8] 2025-08-06 17:38:52,417 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
949:[hw_device_mgr-8] 2025-08-06 17:38:52,419 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
950:[hw_device_mgr-8] 2025-08-06 17:38:52,419 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
951:[hw_device_mgr-8] 2025-08-06 17:38:57,717 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
952:[hw_device_mgr-8] 2025-08-06 17:38:57,718 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
953:[hw_device_mgr-8] 2025-08-06 17:38:57,718 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
954:[hw_device_mgr-8] 2025-08-06 17:39:02,917 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
955:[hw_device_mgr-8] 2025-08-06 17:39:02,919 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
956:[hw_device_mgr-8] 2025-08-06 17:39:02,919 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
957:[hw_device_mgr-8] 2025-08-06 17:39:08,117 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
958:[hw_device_mgr-8] 2025-08-06 17:39:08,118 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
959:[hw_device_mgr-8] 2025-08-06 17:39:08,119 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
960:[hw_device_mgr-8] 2025-08-06 17:39:12,218 [ERROR]ZA_E7.820.003@(0,6,0): Timeout (30s):  device params unset
961:[hw_device_mgr-8] 2025-08-06 17:39:12,218 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
962:[hw_device_mgr-8] 2025-08-06 17:39:12,218 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
963:[hw_device_mgr-8] 2025-08-06 17:39:17,417 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
964:[hw_device_mgr-8] 2025-08-06 17:39:17,419 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
965:[hw_device_mgr-8] 2025-08-06 17:39:17,419 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
966:[hw_device_mgr-8] 2025-08-06 17:39:22,717 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
967:[hw_device_mgr-8] 2025-08-06 17:39:22,718 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
968:[hw_device_mgr-8] 2025-08-06 17:39:22,719 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
969:[hw_device_mgr-8] 2025-08-06 17:39:28,017 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
970:[hw_device_mgr-8] 2025-08-06 17:39:28,018 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
971:[hw_device_mgr-8] 2025-08-06 17:39:28,018 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
972:[hw_device_mgr-8] 2025-08-06 17:39:33,217 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
973:[hw_device_mgr-8] 2025-08-06 17:39:33,218 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
974:[hw_device_mgr-8] 2025-08-06 17:39:33,219 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
975:[hw_device_mgr-8] 2025-08-06 17:39:38,517 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
976:[hw_device_mgr-8] 2025-08-06 17:39:38,519 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
977:[hw_device_mgr-8] 2025-08-06 17:39:38,519 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
978:[hw_device_mgr-8] 2025-08-06 17:39:42,418 [ERROR]ZA_E7.820.003@(0,6,0): Timeout (30s):  device params unset
979:[hw_device_mgr-8] 2025-08-06 17:39:42,418 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
980:[hw_device_mgr-8] 2025-08-06 17:39:42,418 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
981:[hw_device_mgr-8] 2025-08-06 17:39:47,717 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
982:[hw_device_mgr-8] 2025-08-06 17:39:47,720 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
983:[hw_device_mgr-8] 2025-08-06 17:39:47,721 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
984:[hw_device_mgr-8] 2025-08-06 17:39:53,017 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
985:[hw_device_mgr-8] 2025-08-06 17:39:53,018 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
986:[hw_device_mgr-8] 2025-08-06 17:39:53,018 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
993:[hw_device_mgr-8] 2025-08-06 17:39:54,718 [INFO]ZA_device_mgr: State command set:  'start'
994:[hw_device_mgr-8] 2025-08-06 17:39:54,718 [WARNING]ZA_device_mgr: Unable to honor start_command command: Cannot trigger event start_command because the onbeforestart_command handler returns False
995:[hw_device_mgr-8] 2025-08-06 17:39:58,217 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
996:[hw_device_mgr-8] 2025-08-06 17:39:58,219 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
997:[hw_device_mgr-8] 2025-08-06 17:39:58,220 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
998:[hw_device_mgr-8] 2025-08-06 17:40:03,517 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
999:[hw_device_mgr-8] 2025-08-06 17:40:03,518 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
1000:[hw_device_mgr-8] 2025-08-06 17:40:03,518 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
1003:[hw_device_mgr-8] 2025-08-06 17:40:08,817 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
1004:[hw_device_mgr-8] 2025-08-06 17:40:08,818 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
1005:[hw_device_mgr-8] 2025-08-06 17:40:08,818 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
1006:[hw_device_mgr-8] 2025-08-06 17:40:12,718 [ERROR]ZA_E7.820.003@(0,6,0): Timeout (30s):  device params unset
1007:[hw_device_mgr-8] 2025-08-06 17:40:12,718 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
1008:[hw_device_mgr-8] 2025-08-06 17:40:12,718 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False
1009:^A[hw_device_mgr-8] 2025-08-06 17:40:17,917 [ERROR]ZA_device_mgr: Timeout (5s):  device params unset (ZA_E7.820.003@(0,6,0))
1010:[hw_device_mgr-8] 2025-08-06 17:40:17,919 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
1011:[hw_device_mgr-8] 2025-08-06 17:40:17,919 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False

We never see the drives get sent the Switch On Enabled command. Is this being blocked by the hw_device_mgr entering a fault state?

(ZA_IS620N@(0,0,0),ZA_IS620N@(0,1,0),ZA_IS620N@(0,2,0),ZA_IS620N@(0,3,0),ZA_IS620N@(0,4,0),ZA_IS620N@(0,5,0)); device params unset (ZA_E7.820.003@(0,6,0))
[hw_device_mgr-8] 2025-08-06 17:37:46,619 [WARNING]ZA_device_mgr: Manager/device fault; entering fault state
[hw_device_mgr-8] 2025-08-06 17:37:46,619 [WARNING]ZA_device_mgr: Unable to honor fault_command command: Cannot trigger event fault_command because the onbeforefault_command handler returns False

Could this be related to the auxiliary eStop? I looked at some of the ros2 configuration files but couldn’t see where this estop was defined.

As always, any help is much appreciated. It feels really close now

Cheers,
Darran.

Attached is the full main log for reference:
paste1.txt.nc (133.1 KB)

John_Morris · August 7, 2025, 6:30pm

Ah ha, this problem IS a difference with the early machine, which has the additional I/O module. I’m remote and rebuilt my robot to be more like the later machines without I/O module. I do have one I can put in there and try out. Do you mind these testing cycles I’m putting you through? If not, I think we could probably have you up and running pretty quickly. Cross fingers this will be the last one. I’ll update here when I have a new fix.

futnuh · August 7, 2025, 7:19pm

Mind? Absolutely not. I’m ecstatic that you are helping me with this. Looking forward to trying the next patch.

John_Morris · August 8, 2025, 12:04am

Thanks for being so gracious about the tedious test cycles! It’s more likely we’ll get this working faster with you testing on your side. It would take a little longer to round up someone at HQ to test on the R&D machine with drives like yours, so it’s great that you’re willing.

Anyway, I think we’re close to solving the latest errors. I hooked up an IO module and reproduced your problem. Fix pushed to the PR branch. You’ll have to rebuild the Docker image once again to pick up changes in the external hw_device_mgr repo. Crossing fingers this gets your robot up and running, and you hear that “click”…

futnuh · August 8, 2025, 3:40am

Success!! The brakes release and I can move the robot via Rviz using the default ompl planner. Thanks so much for doing this.

After an hour of play, I have started to chart out a path forward.

Bring up Rviz separately from the robot. Eventually I’ll probably run Rviz on a separate machine or even write my own front-end. For now, I just want to defer running Rviz until the robot proper is fully launched.
While I can monitor and set the digital io via ros2 commands, it would be nice to have an Rviz pane that shows this pins and also toggling the state.
Add the auxiliary eStop back (/din16) for safety. (Was this previously handled through the pathpilot front end?)
I’d like to install moveit_servo which I believe is part of the open source moveit2, yes? One benefit right away would be teleoperation of the robot with an xbox controller or gamepad (using the ros2 joy package).
Add “warehouse” storage so that I can save robot states and trajectories. Probably sqlite to start.
Add a Stereolabs Zed mini to the mix and use this for object detection and avoidance. (I’ve used this hardware before with success, albeit not within a ros context.)

I’d be very interested to know if you’ve done either steps 4 or 5 and/or are aware of any stumbling blocks - or worse - dead ends?

Cheers,
Darran.

John_Morris · August 8, 2025, 4:29pm

Awesome, I’m so glad, and grateful for your help fixing support for the IO module.

As for your points:

ros2 launch za6_bringup bringup.launch use_rviz:=false
I would love a PR.
Looks like I missed that important piece when I did the port. The code from the ROS 1 stack is here (again, I would love a PR!): tormach_za_ros1/src/tormach/za6/za6_hardware/src/hal_plumber/hal_io_plumber.py at noetic · tormach/tormach_za_ros1 · GitHub
Yes moveit_servo is part of moveit2; install with apt-get install ros-humble-moveit-servo. We can discuss a way to get that built into your Docker image to make it more convenient, but there’s nothing in the repo that works “out of the box” for that purpose right now.
In za6_moveit_config/package.xml I see a dependency on warehouse_ros_sqlite, and warehouse_ros_sqlite shows it’s installed. I haven’t used it before, though.
Ooh! At IMTS one of our team did a milling machine tending demo using a YDLIDAR for detecting work pieces placed on a table. It worked great. I’d love to hear how your project progresses, if you’re willing to keep me posted. https://www.youtube.com/watch?v=trBNuH-SAxQ

I think everything you want to do in 1-5 is quite possible, and guessing your experience level, no major challenges. You are, however, in blue water here, and we’ll be able to offer only minimal support: the ROS 2 port is provided “as is”.

OTOH we love customers like you who want to use our tools in ways we didn’t anticipate, and I’m personally interested in improving the ROS 2 stack, a pet project of mine. We also love customers who share their progress with the community, and if you wanted to do that, we’d be highly motivated to make you successful!

nathan_lambert · September 15, 2025, 6:44pm

Good afternoon.

I’m trying to upgrade our ZA-6 arm from ROS1 to ROS2 and I’m following the provided Readme file. I have successfully built the docker container however, I am currently unable to get the bring up program to run successfully. Initially RVIS2 and MoveIt2 both launch but shortly afterwards everything crashes.

I get the following describing from Terminal:

[rviz2-7] at line 253 in /opt/ros/humble/include/class_loader/class_loader/class_loader_core.hpp
[rviz2-7] [ERROR 1757959440.324647547] [moveit_ros_visualization.motion_planning_frame]: Action server: /recognize_objects not available (MotionPlanningFrame() at ./motion_planning_rviz_plugin/src/motion_planning_frame.cpp:214)
[rviz2-7] [INFO 1757959440.343410611] [moveit_ros_visualization.motion_planning_frame]: MoveGroup namespace changed: / → . Reloading params. (enable() at ./motion_planning_rviz_plugin/src/motion_planning_frame.cpp:597)
[rviz2-7] [INFO 1757959440.837591249] [moveit_rdf_loader.rdf_loader]: Loaded robot model in 0.418551 seconds (RDFLoader() at ./rdf_loader/src/rdf_loader.cpp:80)
[rviz2-7] [INFO 1757959440.837640616] [moveit_robot_model.robot_model]: Loading robot model ‘za6’… (buildModel() at ./robot_model/src/robot_model.cpp:89)
[rviz2-7] [INFO 1757959440.837658642] [moveit_robot_model.robot_model]: No root/virtual joint specified in SRDF. Assuming fixed joint (constructJointModel() at ./robot_model/src/robot_model.cpp:959)
[rviz2-7] [INFO 1757959440.881353184] [moveit_ros.planning_scene_monitor.planning_scene_monitor]: Starting planning scene monitor (startSceneMonitor() at ./planning_scene_monitor/src/planning_scene_monitor.cpp:1108)
[rviz2-7] [INFO 1757959440.882763337] [moveit_ros.planning_scene_monitor.planning_scene_monitor]: Listening to ‘/monitored_planning_scene’ (startSceneMonitor() at ./planning_scene_monitor/src/planning_scene_monitor.cpp:1116)
[rviz2-7] [INFO 1757959441.065819939] [interactive_marker_display_94910103613952]: Connected on namespace: /rviz_moveit_motion_planning_display/robot_interaction_interactive_marker_topic (updateStatus() at ./src/interactive_marker_client.cpp:406)
[rviz2-7] [INFO 1757959441.073619915] [moveit_ros_visualization.motion_planning_frame]: group manipulator (changePlanningGroupHelper() at ./motion_planning_rviz_plugin/src/motion_planning_frame.cpp:379)
[rviz2-7] [INFO 1757959441.073647185] [moveit_ros_visualization.motion_planning_frame]: Constructing new MoveGroup connection for group ‘manipulator’ in namespace ‘’ (changePlanningGroupHelper() at ./motion_planning_rviz_plugin/src/motion_planning_frame.cpp:382)
[rviz2-7] [INFO 1757959441.086368755] [interactive_marker_display_94910103613952]: Sending request for interactive markers (updateStatus() at ./src/interactive_marker_client.cpp:406)
[rviz2-7] [INFO 1757959441.100797481] [move_group_interface]: Ready to take commands for planning group manipulator. (MoveGroupInterfaceImpl() at ./move_group_interface/src/move_group_interface.cpp:185)
[rviz2-7] [INFO 1757959441.102787341] [moveit_ros_visualization.motion_planning_frame]: group manipulator (changePlanningGroupHelper() at ./motion_planning_rviz_plugin/src/motion_planning_frame.cpp:379)
[rviz2-7] [INFO 1757959441.117618625] [interactive_marker_display_94910103613952]: Service response received for initialization (updateStatus() at ./src/interactive_marker_client.cpp:406)
[spawner-2] [WARN 1757959447.262850445] [spawner_joint_trajectory_controller]: Failed getting a result from calling /controller_manager/list_controllers in 10.0. (Attempt 1 of 3.) (service_caller() at /opt/ros/humble/local/lib/python3.10/dist-packages/controller_manager/controller_manager_services.py:144)
[hal_mgr-1] halcmd: cant connect to rtapi_app: -1 (uri= uuid=67273f03-a5c8-4ac7-b1cd-e74fb8736219): rtapi_rpc(): reply timeout
[hal_mgr-1]
[spawner-4] [WARN 1757959447.315061131] [spawner_joint_state_broadcaster]: Failed getting a result from calling /controller_manager/list_controllers in 10.0. (Attempt 1 of 3.) (service_caller() at /opt/ros/humble/local/lib/python3.10/dist-packages/controller_manager/controller_manager_services.py:144)
[spawner-3] [WARN 1757959447.318067579] [spawner_streaming_controller]: Failed getting a result from calling /controller_manager/list_controllers in 10.0. (Attempt 1 of 3.) (service_caller() at /opt/ros/humble/local/lib/python3.10/dist-packages/controller_manager/controller_manager_services.py:144)
[hal_mgr-1] E: 25-09-15 13:04:07 [5952]dangling ‘DEALER’ socket created at hal/utils/halcmd_rtapiapp.cc:290
[hal_mgr-1] Traceback (most recent call last):
[hal_mgr-1] File “/opt/ros/humble/lib/hal_hw_interface/hal_mgr”, line 38, in
[hal_mgr-1] HalMgr(sys.argv).main()
[hal_mgr-1] File “/opt/ros/humble/local/lib/python3.10/dist-packages/hal_hw_interface/ros_hal_component.py”, line 153, in main
[hal_mgr-1] self.setup_component()
[hal_mgr-1] File “/opt/ros/humble/local/lib/python3.10/dist-packages/hal_hw_interface/hal_mgr.py”, line 23, in setup_component
[hal_mgr-1] self.start_realtime()
[hal_mgr-1] File “/opt/ros/humble/local/lib/python3.10/dist-packages/hal_hw_interface/hal_mgr.py”, line 59, in start_realtime
[hal_mgr-1] subprocess.check_call([“realtime”, “start”], env=env)
[hal_mgr-1] File “/usr/lib/python3.10/subprocess.py”, line 369, in check_call
[hal_mgr-1] raise CalledProcessError(retcode, cmd)
[hal_mgr-1] subprocess.CalledProcessError: Command ‘[‘realtime’, ‘start’]’ returned non-zero exit status 1.
[ERROR] [hal_mgr-1]: process has died [pid 5835, exit code 1, cmd ‘/opt/ros/humble/lib/hal_hw_interface/hal_mgr --ros-args -r __node:=hal_mgr --params-file /tmp/launch_params_exq6nwal’].
[INFO] [launch.user]: hal_mgr shutting down: Exited
[WARNING] [hal_hw_interface.launch.hal_ordered_action]: Canceling handler <HalFiles ‘hal_files’>
[WARNING] [hal_hw_interface.launch.hal_ordered_action]: Canceling handler <HalUserNode ‘drive_state’>
[WARNING] [hal_hw_interface.launch.hal_ordered_action]: Canceling handler <HalUserNode ‘hal_io’>
[WARNING] [hal_hw_interface.launch.hal_ordered_action]: Canceling handler <HalUserNode ‘hw_device_mgr’>
[WARNING] [hal_hw_interface.launch.hal_ordered_action]: Canceling handler <HalRTNode ‘hal_control_node’>
[INFO] [rviz2-7]: sending signal ‘SIGINT’ to process[rviz2-7]
[INFO] [move_group-6]: sending signal ‘SIGINT’ to process[move_group-6]
[INFO] [robot_state_publisher-5]: sending signal ‘SIGINT’ to process[robot_state_publisher-5]
[INFO] [spawner-4]: sending signal ‘SIGINT’ to process[spawner-4]
[INFO] [spawner-3]: sending signal ‘SIGINT’ to process[spawner-3]
[INFO] [spawner-2]: sending signal ‘SIGINT’ to process[spawner-2]
[rviz2-7] [INFO 1757959448.751018741] [rclcpp]: signal_handler(signum=2) (signal_handler() at ./src/rclcpp/signal_handler.cpp:71)
[move_group-6] [INFO 1757959448.752495677] [rclcpp]: signal_handler(signum=2) (signal_handler() at ./src/rclcpp/signal_handler.cpp:71)
[robot_state_publisher-5] [INFO 1757959448.753863291] [rclcpp]: signal_handler(signum=2) (signal_handler() at ./src/rclcpp/signal_handler.cpp:71)
[rviz2-7] [WARN 1757959448.765237154] [interactive_marker_display_94910103613952]: Server not available while running, resetting (updateStatus() at ./src/interactive_marker_client.cpp:409)
[move_group-6] [INFO 1757959448.796721120] [moveit.ros_planning_interface.moveit_cpp]: Deleting MoveItCpp (~MoveItCpp() at ./moveit_cpp/src/moveit_cpp.cpp:88)
[move_group-6] [INFO 1757959448.797445853] [moveit_ros.planning_scene_monitor.planning_scene_monitor]: Stopped publishing maintained planning scene. (stopPublishingPlanningScene() at ./planning_scene_monitor/src/planning_scene_monitor.cpp:394)
[move_group-6] [INFO 1757959448.799904658] [moveit_ros.planning_scene_monitor.planning_scene_monitor]: Stopping world geometry monitor (stopWorldGeometryMonitor() at ./planning_scene_monitor/src/planning_scene_monitor.cpp:1241)
[move_group-6] [INFO 1757959448.801250927] [moveit_ros.planning_scene_monitor.planning_scene_monitor]: Stopping planning scene monitor (stopSceneMonitor() at ./planning_scene_monitor/src/planning_scene_monitor.cpp:1124)
[move_group-6] Warning: class_loader.ClassLoader: SEVERE WARNING!!! Attempting to unload library while objects created by this loader exist in the heap! You should delete your objects before attempting to unload the library or destroying the ClassLoader. The library will NOT be unloaded.
[move_group-6] at line 127 in ./src/class_loader.cpp
[rviz2-7]
[rviz2-7] >>> [rcutils|error_handling.c:108] rcutils_set_error_state()
[rviz2-7] This error state is being overwritten:
[rviz2-7]
[rviz2-7] ‘rcl node’s context is invalid, at ./src/rcl/node.c:428’
[rviz2-7]
[rviz2-7] with this new error message:
[rviz2-7]
[rviz2-7] ‘publisher’s context is invalid, at ./src/rcl/publisher.c:389’
[rviz2-7]
[rviz2-7] rcutils_reset_error() should be called after error handling to avoid this.
[rviz2-7] <<<
[rviz2-7] [INFO 1757959448.885335528] [moveit_ros.planning_scene_monitor.planning_scene_monitor]: Stopping planning scene monitor (stopSceneMonitor() at ./planning_scene_monitor/src/planning_scene_monitor.cpp:1124)
[move_group-6] Stack trace (most recent call last):
[move_group-6] #16 Object “”, at 0xffffffffffffffff, in
[move_group-6] #15 Object “/opt/ros/humble/lib/moveit_ros_move_group/move_group”, at 0x56367d47f724, in
[move_group-6] #14 Object “/usr/lib/x86_64-linux-gnu/libc.so.6”, at 0x7fd4ea73be3f, in __libc_start_main
[move_group-6] #13 Object “/usr/lib/x86_64-linux-gnu/libc.so.6”, at 0x7fd4ea73bd8f, in
[move_group-6] #12 Object “/opt/ros/humble/lib/moveit_ros_move_group/move_group”, at 0x56367d47e65b, in
[move_group-6] #11 Object “/opt/ros/humble/lib/moveit_ros_move_group/move_group”, at 0x56367d480709, in
[move_group-6] #10 Object “/opt/ros/humble/lib/libmoveit_cpp.so.2.5.9”, at 0x7fd4eb09f876, in moveit_cpp::MoveItCpp::~MoveItCpp()
[move_group-6] #9 Object “/opt/ros/humble/lib/libmoveit_cpp.so.2.5.9”, at 0x7fd4eb09dba9, in
[move_group-6] #8 Object “/opt/ros/humble/lib/libmoveit_trajectory_execution_manager.so.2.5.9”, at 0x7fd4ea431495, in trajectory_execution_manager::TrajectoryExecutionManager::~TrajectoryExecutionManager()
[move_group-6] #7 Object “/opt/ros/humble/lib/libmoveit_trajectory_execution_manager.so.2.5.9”, at 0x7fd4ea4443a9, in
[move_group-6] #6 Object “/opt/ros/humble/lib/librclcpp.so”, at 0x7fd4eacfdd8c, in rclcpp::Node::~Node()
[move_group-6] #5 Object “/opt/ros/humble/lib/librclcpp.so”, at 0x7fd4eacfdd2e, in rclcpp::Node::~Node()
[move_group-6] #4 Object “/opt/ros/humble/lib/librclcpp.so”, at 0x7fd4eacd8db9, in
[move_group-6] #3 Object “/opt/ros/humble/lib/librclcpp.so”, at 0x7fd4ead073d9, in
[move_group-6] #2 Object “/opt/ros/humble/lib/librclcpp.so”, at 0x7fd4ead07320, in rclcpp::node_interfaces::NodeBase::~NodeBase()
[move_group-6] #1 Object “/opt/ros/humble/lib/librclcpp.so”, at 0x7fd4eacd8db9, in
[move_group-6] #0 Object “/opt/ros/humble/lib/librclcpp.so”, at 0x7fd4eacddc71, in rclcpp::CallbackGroup::~CallbackGroup()
[move_group-6] Segmentation fault (Address not mapped to object [0x7fd4e08dc798])
[INFO] [robot_state_publisher-5]: process has finished cleanly [pid 5843]
[ERROR] [move_group-6]: process has died [pid 5845, exit code -11, cmd ‘/opt/ros/humble/lib/moveit_ros_move_group/move_group --ros-args --params-file /tmp/launch_params_i3bnzc_7 --params-file /tmp/launch_params_qm844pdz’].
[move_group-6]
[ERROR] [rviz2-7]: process has died [pid 5855, exit code -11, cmd ‘/opt/ros/humble/lib/rviz2/rviz2 -d /home/pathpilot/tormach_za_ros2_ws/install/za6_moveit_config/share/za6_moveit_config/config/moveit.rviz --ros-args --params-file /tmp/launch_params_0n5msyil --params-file /tmp/launch_params_twn9v7wy’].
[ERROR] [spawner-4]: process[spawner-4] failed to terminate ‘5’ seconds after receiving ‘SIGINT’, escalating to ‘SIGTERM’
[ERROR] [spawner-3]: process[spawner-3] failed to terminate ‘5’ seconds after receiving ‘SIGINT’, escalating to ‘SIGTERM’
[ERROR] [spawner-2]: process[spawner-2] failed to terminate ‘5’ seconds after receiving ‘SIGINT’, escalating to ‘SIGTERM’
[INFO] [spawner-4]: sending signal ‘SIGTERM’ to process[spawner-4]
[INFO] [spawner-3]: sending signal ‘SIGTERM’ to process[spawner-3]
[INFO] [spawner-2]: sending signal ‘SIGTERM’ to process[spawner-2]
[ERROR] [spawner-4]: process has died [pid 5841, exit code -15, cmd ‘/opt/ros/humble/lib/controller_manager/spawner --controller-manager-timeout=180 --switch-timeout=180 joint_state_broadcaster --ros-args --params-file /tmp/launch_params__r2n4hrl’].
[ERROR] [spawner-3]: process has died [pid 5839, exit code -15, cmd ‘/opt/ros/humble/lib/controller_manager/spawner --controller-manager-timeout=180 --inactive streaming_controller --ros-args --params-file /tmp/launch_params_6v4oowye’].
[ERROR] [spawner-2]: process has died [pid 5837, exit code -15, cmd ‘/opt/ros/humble/lib/controller_manager/spawner --controller-manager-timeout=180 --switch-timeout=180 joint_trajectory_controller --ros-args --params-file /tmp/launch_params_jg42dgg1’].

Any assistance would be greatly appreciated!