SUMMER2021_TASK-3

Task description:-Live Streaming Video Chat App without voice using cv2 module of Python

Hello Guys!! In this article we will see how we can integrate OpenCV library with webcam, so to live stream.

Some Basics we need to keep in mind before starting:

1. While transferring data over the network, the data must be in bytes. In this task we are going to use Python as a language for Live Streaming. So, from transferring data perspective we have to convert Python Objects into stream of bytes before sending to the receiver.
2. So in Python we have Pickle library by which we can convert Python Objects to stream of bytes. As video is nothing but the continuous clicking of pictures, and from computer perspective images are just numbers.
3. In this task we will see how to use cv2 module to connect to the webcam, so we can click pictures, and when pictures are continuous clicked it is called video.
4. So here our single clicked photo is a Python Object. Before we transfer it to network we have to convert it into stream of bytes.
5. When we click our picture it is stored in the form of arrays in machines.
6. By using dumps() function available in pickle , our image of type numpy.ndarray gets converted to bytes.

When I started this and created a server and receiver python files.

1. Imported cv2 library, socket and pickle library

It was an attempt to transfer photo over the network

sender.py

import socket, cv2, pickle
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host_name = socket.gethostname()
host_ip = socket.gethostbyname(host_name)
port = 1234#Socket Creation
socket_address = (host_ip,port)
#socket binding
server_socket.bind(socket_address)server_socket.listen(1)
import pickle
cap = cv2.VideoCapture(0)ret, photo = cap.read()
    
# This is in bytes now
file_to_transfer = pickle.dumps(photo)
    
cv2.imwrite("my.jpg" , photo)
cv2.imshow("my photo" , photo )
cv2.waitKey()
cv2.destroyAllWindows()
client_socket.sendall(file_to_transfer)

receiver.py

import socket,pickle
client_socket = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
host_ip = '192.168.99.1'
port = 9999
client_socket.connect((host_ip,port))
packet_data = client_socket.recv(4*1024)
data = packet_data
data = pickle.loads(data)
cv2.imshow("Photo_Received",data)

But the above codes threw this error.

Error at the receiver end

loads() function available in pickle module helps in getting the image back that we have converted into stream of bytes. One of the reasons for this error was due to the buffer size that I specified by using recv() function. This function tried to receive 4*1024 bytes from the connection established using socket but failed, it might be due to the size of the bytes that server was trying to send to receiver exceeded the buffer limit.

So what’s the solution for this thing. For this problem to get resolved here comes the library named struct in picture. What does this library do and what various functions it offers to resolve the above issue? We will check out.

To demonstrate the use of this library let’s try to create an example that will help us to better understand it.

Plan

1. Click a picture.
2. Check the datatype of this image
3. Convert this image into bytes using pickle module
4. See what this bytes looks like

import cv2cap  =   cv2.VideoCapture(0)
ret ,  photo  =  cap.read()
cv2.imwrite("my1.jpg" , photo)cap.release()

#Converting this image to bytes format 
import pickle
photo_serialize=pickle.dumps(photo)

Data type changed after applying dumps() function

Now let’s have a look inside photo_serialize

Now here at this point the receiver end was not able to judge how much buffer size is required. What we can do now? If by someway we can tell the receiver this is the size of the data and you need this much storage to store the data. Now this is the right time to get started with struct module in Python.

import struct
size = struct.calcsize("Q")

calcsize() function in struct module : Here “Q” is the value you can pass in the arguments so that it provides 8 bytes. Similarly there are many other values you can pass. Below is the link to check other values.

struct — Interpret bytes as packed binary data — Python 3.9.5 documentation

Source code: Lib/struct.py Note By default, the result of packing a given C struct includes pad bytes in order to…

docs.python.org

So, now there is another thing that we are going to learn. When we will packed the data using the pack() function available in struct module, the first 8 bytes corresponds to the size of the file . How I can say this ? Below is the example.

Now I will add the serialized photo that is in bytes with the pack() function that contains the information regrading the size in hexadecimal format.

In the above picture, the first 8 bytes stored the information regrading the size and from \x80 onwards we have the photo that we converted into bytes.

Now let’s try again to do streaming of video.

sender.py

import socket, cv2, pickle,struct

# Socket Create
server_socket = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
host_name  = socket.gethostname()
host_ip = socket.gethostbyname(host_name)
print('HOST IP:',host_ip)
port = 9999
socket_address = (host_ip,port)

# Socket Bind
server_socket.bind(socket_address)

# Socket Listen
server_socket.listen(1)
print("LISTENING AT:",socket_address)

# Socket Accept
while True:
	client_socket,addr = server_socket.accept()
	print('GOT CONNECTION FROM:',addr)
	if client_socket:
		vid = cv2.VideoCapture(0)
		
		while(vid.isOpened()):
			ret,image = vid.read()
			img_serialize = pickle.dumps(image)
			message = struct.pack("Q",len(img_serialize))+img_serialize
			client_socket.sendall(message)
			
			cv2.imshow('VIDEO FROM SERVER',image)
			key = cv2.waitKey(10) 
			if key ==13:
                             client_socket.close()

view rawsender.py hosted with ❤ by GitHub

receiver.py

import socket,cv2, pickle,struct

# create socket
client_socket = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
#  server ip address here
host_ip = '192.168.99.1' 
port = 9999
client_socket.connect((host_ip,port)) 
data = b""
metadata_size = struct.calcsize("Q")
while True:
	while len(data) < metadata_size:
		packet = client_socket.recv(4*1024) 
		if not packet: break
		data += packet
	packed_msg_size = data[:metadata_size]
	data = data[metadata_size:]
	msg_size = struct.unpack("Q",packed_msg_size)[0]
	
	while len(data) < msg_size:
		data += client_socket.recv(4*1024)
	frame_data = data[:msg_size]
	data  = data[msg_size:]
	frame = pickle.loads(frame_data)
	cv2.imshow("RECEIVING VIDEO",frame)
	key = cv2.waitKey(10) 
	if key  == 13:
		break
client_socket.close()

view rawreceiver.py hosted with ❤ by GitHub

This time when we run the server.py and after it we try to run receiver.py, pictures are transferring at a good speed and it seems that live video streaming is going.

Now let’s convert it into video chat app in which we have two clients. Client A and Client B. In this case 4 windows will show up.

This setup you can perform if you have 2 laptops using the Wi-Fi.

These will be:

1. Client A sending to Client B
2. Client B receiving from Client A
3. Client B sending to Client A
4. Client A receiving from Client B

In this case we will need to use the concept of multi-threading. We have to start 2 threads in Client A and Client B simultaneously. One thread will send the video and other will receive the video.

Below is the source code for Client A.

import socket, cv2, pickle,struct, threading, time

# Socket Create
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)

# Socket Accept
def sender():
	time.sleep(15)
	host_name  = socket.gethostname()
	host_ip = socket.gethostbyname(host_name)
	print('HOST IP:',host_ip)
	port = 9999
	socket_address = (host_ip,port)
	# Socket Bind
	s.bind(socket_address)
	# Socket Listen
	s.listen(5)
	print("LISTENING AT:",socket_address)
	while True:
		client_socket,addr = s.accept()
		print('GOT CONNECTION FROM:',addr)
		if client_socket:
			vid = cv2.VideoCapture(0)
		
			while(vid.isOpened()):
				ret,image = vid.read()
				img_serialize = pickle.dumps(image)
				message = struct.pack("Q",len(img_serialize))+img_serialize
				client_socket.sendall(message)
			
				cv2.imshow('VIDEO FROM SERVER',image)
				key = cv2.waitKey(10) 
				if key ==13:
					client_socket.close()

def connect_server():
	host_ip = '192.168.99.1' 
	port = 1234
	s.connect((host_ip,port)) 
	data = b""
	metadata_size = struct.calcsize("Q")
	while True:
		while len(data) < metadata_size:
			packet = s.recv(4*1024) 
			if not packet: break
			data+=packet
		packed_msg_size = data[:metadata_size]
		data = data[metadata_size:]
		msg_size = struct.unpack("Q",packed_msg_size)[0]
	
		while len(data) < msg_size:
			data += s.recv(4*1024)
		frame_data = data[:msg_size]
		data  = data[msg_size:]
		frame = pickle.loads(frame_data)
		cv2.imshow("RECEIVING VIDEO",frame)
		key = cv2.waitKey(10) 
		if key  == 13:
			break
	s.close()


x1 = threading.Thread(target=sender)

x2 = threading.Thread(target=connect_server)

# start a thread
x1.start()
x2.start()

Source code for Client B

import socket,cv2, pickle,struct, time
import threading
# create socket
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
def connect_server():
	time.sleep(15)
	host_ip = '192.168.99.1' 
	port = 9999
	s.connect((host_ip,port)) 
	data = b""
	metadata_size = struct.calcsize("Q")
	while True:
		while len(data) < metadata_size:
			packet = s.recv(4*1024) 
			if not packet: break
			data+=packet
		packed_msg_size = data[:metadata_size]
		data = data[metadata_size:]
		msg_size = struct.unpack("Q",packed_msg_size)[0]
	
		while len(data) < msg_size:
			data += s.recv(4*1024)
		frame_data = data[:msg_size]
		data  = data[msg_size:]
		frame = pickle.loads(frame_data)
		cv2.imshow("RECEIVING VIDEO",frame)
		key = cv2.waitKey(10) 
		if key  == 13:
			break
	s.close()

def sender():
	host_name  = socket.gethostname()
	host_ip = socket.gethostbyname(host_name)
	print('HOST IP:',host_ip)
	port = 1234
	socket_address = (host_ip,port)
	# Socket Bind
	s.bind(socket_address)
	# Socket Listen
	s.listen(5)
	print("LISTENING AT:",socket_address)
	while True:
		client_socket,addr = s.accept()
		print('GOT CONNECTION FROM:',addr)
		if client_socket:
			vid = cv2.VideoCapture(1)
		
			while(vid.isOpened()):
				ret,image = vid.read()
				img_serialize = pickle.dumps(image)
				message = struct.pack("Q",len(img_serialize))+img_serialize
				client_socket.sendall(message)
			
				cv2.imshow('VIDEO FROM SERVER',image)
				key = cv2.waitKey(10) 
				if key ==13:
					client_socket.close()


x1 = threading.Thread(target=connect_server)
x2 = threading.Thread(target=sender)


x1.start()
x2.start()

Thank you for reading.