If the IM program crashes while using your camera, this would be because either the program is faulty in some way, or your PC is being overworked.
If you think that the IM program is faulty, try the latest (or a previous) version of the software to see if the situation improves.
You can easily see if your PC is being overworked by calling up the Task Manager (Ctrl-Shift-Esc), and note the CPU Usage in the graph. To reduce the CPU load, try reducing the image size and/or the FPS of the transmitted and received "signals".
Be aware that when you are in a video conversation, your PC is rendering two audio and two video streams...not just one...as it would be when you are just using QuickCam to record a video. That's twice the work that it would "normally" perform in this situation.
Next, you'll need to find out if it is the audio stream or the video stream that is causing the crash to occur. See if you can successfully initiate each of these "conversations" separately...to help narrow down the cause of the crashing. For instance, if you can successfully initiate a "video only" conversation, but the IM crashes once you add audio, then you will know that the problem is somehow related to the addition of the audio stream. You would then need to initiate an "audio only" conversation to confirm this hypothesis.
"Webcam in use" message appears: You will see this message if you are accessing the camera in QuickCapture prior to using your Instant Messenger program.
Since only one "video aware" application can use the camera at a time, any and all other subsequent attempts to "connect" to it will fail.
Exception: You are using third-party "video cloning" software that will allow multiple applications to see and use the camera simultaneously.
Technically speaking, the appearance of this message means that another application has a handle open to the device.