Soccer Vision Msgs, and next steps!

Ideas & Discussion
, , , ,
#1

Working with Hamburg Bit-bots, we have been developing a ROS package that defines ROS2 interfaces for the Soccer Domain.

The plan is to come up with general ROS2 interfaces that encapsulate teams’ needs, and for teams and leagues to be able to share software. The package aims to be used inter-league, and we appreciate all feedback in using the package.

Recently, we finished the first prototype of Soccer Vision Msgs, a package containing ROS msg definitions that can be used as the output for vision algorithms specifically for the soccer domain.

As the next step with adopting the msgs, I think we need to write packages to:

  • do the visual recognition and publishing of the msgs on appropriate topics
  • visualize the detected objects, perhaps as an overlay on the original image. Idea for this is to:
    • Write a node that subscribes to the vision msgs and the image, draws the overlays onto the image, then republishes the image on another topic.
    • Have the ability to enable/disable overlays with parameters for that node, that can be changed during run time.

@jaagut @flova What are your thoughts on the next steps?

#2

regarding the message package:
I looked at it and did not find anything missing for now. Probably we will find any issues when we start to use it. Good work :+1:

regarding the visualization:
Yes, this would be helpful and I would be in favor to have the implementation like you proposed it. We have something similar integrated in our vision code that could be used as a basis to implement this.

regarding the recognition:
I’m not sure if it makes sense to have one exact package for this since this is still quiet active research. Maybe it would be easier just to have a list do different recognition implementation that are compatible to the specified message.

regarding further next steps:
I guess the transfer from the 2D image space to the 3D space would be next. Including a software part that does the inverse perspective mapping (we can provide an implementation), message definitions and visualization tools, e.g. a node which publishes corresponding RViz markers to show the objects and one that uses interactive markers to “fake” the objects (we have implementations for both in ROS1 which could be used as a basis)

#3

@bestmann-HBB Welcome to RoboCup Cross-Cutting Discourse and thank you for your post!

Would you be able to edit your post with links to the repositories you’ve mentioned?

It definitely does not make sense to have one package for this, I’d expect teams to have their own package, unless they’re using another teams. Does Bit-Bots have plans on porting their existing vision system to the new msg types some time soon? I think this is an important step to iteratively improve soccer_vision_msgs.

This is a common problem for any robot that has cameras without depth sensors, but I don’t know of any packages that already do this in ROS. If there aren’t any, this could be a good contribution of RoboCup to the wider robotics community!

On my end, I will start working on visualization of soccer_vision_msgs in rviz.

#4

I agree with Marc on the points.

Regarding our Vision, I will most likely start with the migration in the next weeks, as it is one of the first things we would like to port to ros 2. The adaptation of the new messages should be no problem. But it should be expected that our pipeline only detects a subset of the features supported by the new interface.

I also like the idea of a more general implementation for these sorts of projections, maybe with a modular architecture where one can write custom classes for special messages and there is also a set of predefined classes for standard messages. All of them inherit the same backend to do the projection and there the projection planes can be linked to different frames, so that a robot is able to project onto more surfaces than just the floor (e.g. tables or similar things.).

What are you planning regarding the visualization? We are currently drawing the 2d detections onto a debug image that is published to a debug topic. Is this similar to your plans?

1 Like
#5

Our of some reason I cannot edit my post. Here are the links

Drawing debug image: https://github.com/bit-bots/bitbots_vision/blob/adc3e3ad8e256c628966a2cc1d00277903a74f14/bitbots_vision/src/bitbots_vision/vision_modules/debug.py#L5

Inverse perspective mapping: https://github.com/bit-bots/humanoid_league_misc/tree/master/humanoid_league_transform

RViz Markers: https://github.com/bit-bots/humanoid_league_visualization/tree/master/humanoid_league_rviz_marker

RViz Interactive Markers: https://github.com/bit-bots/humanoid_league_visualization/tree/master/humanoid_league_interactive_marker

We don’t know of any package either and I agree that this would be something that is useful in may use cases. I think our current implementation provides all the math for it and would only be necessary to make the software structure more general.

#6

Thanks for the links.

With the image perspective mapping, perhaps we could make use of image_geometry::PinholeCameraModel::projectPixelTo3dRay for generalisation.

We could write a function that finds the intersection between any given plane (the assumption), and the 3d ray returned by projectPixelTo3dRay. For example, the 2d plane for a ball might be a horizontal plane at BALL_RADIUS height.

With the RViz Markers, I think it may good to write separate packages for the different leagues (in different repos), rather than having one package handle all leagues. Different markers would be necessary because the soccer objects are quite different across leagues.

#7

Yes that sounds good.

I don’t think so. Basically the node would publish markers for all types of objects that are specified in the soccer messages. It would only publish markers for the kind of objects it receives information on anyway. Therefore, if you don’t use/recognize some object types, there will be no markers automatically.

#8

We could do this, but in our implementation, we did this with the “raw” vector math, which is pretty straightforward and can be done for many pixels at once using tensor math and libraries like NumPy for Python or Eigen for cpp. This is quite helpful if you want to project an image mask or a whole image to a point cloud efficiently. But I totally agree with your points regarding the planes. We also place the plane with an offset like the BALL_RADIUS.

I agree with Marc on this point. As long as the markers contain most information from the standardized message it should be fine IMO.

#9

Being able to map multiple points effectively is important, and you’re right that calling image_geometry::PinholeCameraModel::projectPixelTo3dRay on every pixel would likely be inefficient.

Which package should we aim to put this functionality in? Should we aim to merge it into image_geometry (which is part of ros-perception/vision_opencv), or start a new package on ros-sports. If the latter, what should the name and purpose of the package be?

I’m going to start developing the packages to draw the overlays of soccer_vision_msgs onto an image.

#10

I think a separate package would be the best fit for this functionality, but it appeals to a wider audience so the ros-sport owner might be confusing. But I think this might also be a chance to promote the ros-sport organization a bit.

I’m going to start developing the packages to draw the overlays of soccer_vision_msgs onto an image.

Nice

I am currently porting our vision to ROS2. It works pretty well now, but it still uses our old messages, because we wanted to change one thing at a time and the downstream nodes need the interface. But we could test the new messages with it on a separate branch, even if our vision only supports a subset of the possible functionalities offered by the new interface.